Skip to content

fix: stop excluding measurement fields whose names contain "id" from metric detection - #4

Open
EAWRCEQA wants to merge 1 commit into
aws-samples:mainfrom
EAWRCEQA:fix-metric-detection-id-substring
Open

fix: stop excluding measurement fields whose names contain "id" from metric detection#4
EAWRCEQA wants to merge 1 commit into
aws-samples:mainfrom
EAWRCEQA:fix-metric-detection-id-substring

Conversation

@EAWRCEQA

Copy link
Copy Markdown

Problem

_detect_metric_field() in agentic_core/schema_inference.py:96 decides whether a numeric column should become an aggregatable metric in the auto-generated semantic layer. It excludes "ID-like" fields with a substring test:

if any(p in name_lower for p in ["_id", "id", "code", "编号", "year", "月", "日"]):
    return False

Because "id" is checked as a substring, it matches any field name that merely contains those two letters. Common measurement columns are silently dropped from metrics: width, humidity, grid_load, candidate_count, and similar. On data upload (POST /api/data/uploadinfer_schema), these valid metrics never make it into the semantic layer, so the assistant can't aggregate them.

Note this is inconsistent with the sibling function _detect_id_field() (same file), which already matches ID patterns precisely via endswith / equality rather than substring.

Reproduction

Loading the module and calling infer_schema on genuine (non-ID, low-cardinality) float measurements:

metrics detected by infer_schema: ['总营收']
  humidity   type=float  is_metric=False is_id=False
  width      type=float  is_metric=False is_id=False
  revenue    type=float  is_metric=True  is_id=False

width and humidity have is_id=False — they are legitimate metrics — yet are excluded purely because their names contain id.

Fix

Match id / _id precisely (standalone name or _id suffix), mirroring _detect_id_field:

if name_lower == "id" or name_lower.endswith("_id"):
    return False
if any(p in name_lower for p in ["code", "编号", "year", "月", "日"]):
    return False

After the fix, the same input yields:

metrics detected by infer_schema: ['平均湿度', '总营收', '平均宽度']
  humidity  is_metric=True
  width     is_metric=True

and real ID columns are still excluded (id, vehicle_id, user_id → not metrics).

Verification

  • python -m py_compile agentic_core/schema_inference.py passes.
  • Ran the module directly (via importlib, so it doesn't pull the package's optional strands import) before and after the change; confirmed the excluded measurement fields are recovered and that id / *_id columns stay excluded.

The change is limited to the name-matching heuristic in _detect_metric_field; the code / 编号 / year / / matches are preserved unchanged.

_detect_metric_field skipped any numeric field whose lowercased name contained
the substring "id", which unintentionally matched common measurement columns
such as width, humidity, grid_load and candidate_count. These aggregatable
metrics were dropped from the auto-generated semantic layer on data upload.

Match "id"/"_id" precisely (standalone name or _id suffix), mirroring the
existing _detect_id_field logic, so real ID columns (id, vehicle_id, user_id)
are still excluded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant