Skip to content

Commit 5bcd099

Browse files
fix(llmobs): prevent dots in evaluation metric labels (#15297)
## Description EVP interprets dots as nested objects (e.g., "field.1" becomes {field: {1: value}}), causing confusion for customers submitting custom evaluations with dots in label names. Add validation to reject labels containing dots and raise a clear error message directing users to use alternative naming conventions. ## Testing Added a unit test to verify the expected behavior. ## Risks None ## Additional Notes JIRA ticket: https://datadoghq.atlassian.net/browse/MLOB-4557
1 parent 1338d6e commit 5bcd099

File tree

3 files changed

+15
-0
lines changed

3 files changed

+15
-0
lines changed

ddtrace/llmobs/_llmobs.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1713,6 +1713,10 @@ def submit_evaluation(
17131713
error = "invalid_metric_label"
17141714
raise ValueError("label must be the specified name of the evaluation metric.")
17151715

1716+
if "." in label:
1717+
error = "invalid_label_value"
1718+
raise ValueError("label value must not contain a '.'.")
1719+
17161720
metric_type = metric_type.lower()
17171721
if metric_type not in ("categorical", "score", "boolean"):
17181722
error = "invalid_metric_type"
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
fixes:
2+
- |
3+
LLM Observability: This fix resolves an issue where evaluation-metric labels containing dots could be interpreted as nested objects by adding validation that rejects such labels and provides a clear error message instructing users to use alternative naming conventions.
4+

tests/llmobs/test_llmobs_service.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1630,6 +1630,13 @@ def test_submit_evaluation_empty_label_raises_error(llmobs, mock_llmobs_logs):
16301630
)
16311631

16321632

1633+
def test_submit_evaluation_label_value_with_a_period_raises_error(llmobs, mock_llmobs_logs):
1634+
with pytest.raises(ValueError, match="label value must not contain a '.'."):
1635+
llmobs.submit_evaluation(
1636+
span={"span_id": "123", "trace_id": "456"}, label="toxicity.0", metric_type="categorical", value="high"
1637+
)
1638+
1639+
16331640
def test_submit_evaluation_incorrect_metric_type_raises_error(llmobs, mock_llmobs_logs):
16341641
with pytest.raises(ValueError, match="metric_type must be one of 'categorical', 'score', or 'boolean'."):
16351642
llmobs.submit_evaluation(

0 commit comments

Comments
 (0)