Match the generic "reference" term as a token, not a bare substring

[cmungall]

Problem

ReferenceURIs.is_reference_uri included the bare substring "reference" in its match patterns, so any URI that merely contained those letters was classified as an authoritative-reference field:

test:user_preference   -> True   (false positive)
ex:dereference         -> True   (false positive)
schema:preferenceOrder -> True   (false positive)

A slot mis-detected as a reference then gets its value passed to _extract_reference_id and fetched, which can surface spurious Could not fetch reference errors on completely unrelated fields.

Note this is intentionally not a plain word-boundary fix: an existing test (test_field_detection.py) pins http://example.org/myReferenceField -> True, i.e. the camelCase Reference token should match. A \breference\b regex (or simply dropping the substring) would break that.

Fix

Match the generic word reference/references as a whole token — splitting the URI on camelCase boundaries and separators (_ - : / #) — while keeping the specific, unambiguous Dublin Core / legacy URIs (dcterms:source, authoritative_reference, …) as substring matches.

Result:

URI	Before	After
dcterms:references	✅	✅
linkml:authoritative_reference	✅	✅
http://example.org/myReferenceField	✅	✅
ex:cross_reference	✅	✅
test:user_preference	❌ flagged	✅ rejected
ex:dereference	❌ flagged	✅ rejected

isReferencedBy-style URIs (token referenced, not reference) are also no longer matched, which is the intended behaviour.

Scope: limited to reference detection, as the bare-substring style for excerpt/title is far less collision-prone.

Tests

• New TestReferenceURITokenBoundaries (test-first; the false-positive cases failed before the fix) covering rejected non-reference terms and preserved genuine references.
• Completed two under-specified mocks in test_title_validation.py that omitted slot_uri (their sibling title mock already set it to None). Real slots always carry slot_uri as str | None; the old pure-substring path only tolerated the stray MagicMock by accident. No assertions changed.

Full non-integration suite: 493 passed. Module doctests, mypy, and ruff clean.

🤖 Generated with Claude Code

[claude]

Claude finished @cmungall's task —— View job

---

PR Code Review ✅

I've completed a comprehensive review of this pull request. This is a well-implemented fix for a subtle but important bug in reference URI detection.

Todo List:

• Read repository setup and guidelines
• Analyze the core implementation changes in field_detection.py
• Review the new test coverage in test_field_detection.py
• Check the test fixes in test_title_validation.py
• Evaluate code quality and best practices
• Assess potential bugs, performance, and security considerations
• Provide comprehensive review feedback

---

Review Summary

This PR successfully fixes a false positive bug in reference URI detection while maintaining backward compatibility and following good software engineering practices.

✅ Strengths

Excellent Problem Analysis & Solution Design

• Root Cause Identification: Correctly identified that bare substring matching of "reference" was causing false positives
• Surgical Fix: The solution is targeted and preserves all existing legitimate functionality
• Backward Compatibility: Maintains compatibility with existing schemas using specific Dublin Core URIs

High-Quality Implementation (src/linkml_reference_validator/field_detection.py)

• Clean Tokenization Logic: The _uri_tokens() function at field_detection.py:40-58 is well-designed:
  • Handles camelCase splitting properly: myReferenceField → ['my', 'reference', 'field']
  • Splits on common URI separators (_, -, :, /, #)
  • Lowercases for case-insensitive matching
• Clear Architecture: Separates concerns by having:
  • MATCH_PATTERNS for specific, unambiguous substring matches
  • TOKEN_TERMS for generic terms that need whole-word matching
• Good Documentation: Comprehensive docstrings with examples demonstrating the fix

Comprehensive Test Coverage

• Test-First Development: New tests in TestReferenceURITokenBoundaries specifically target the bug
• Edge Case Coverage: Tests cover both false positives (should be rejected) and legitimate cases (should still match)
• Parametrized Tests: Good use of pytest parametrization for maintainability
• Real-World Examples: Test cases use realistic URIs that could actually occur

Adherence to Repository Standards

• Follows CLAUDE.md Guidelines: Uses docstrings extensively, avoids unnecessary try/except blocks
• Test-Driven Approach: Aligns with the project's TDD methodology
• Code Quality: Clean, readable code that follows Python best practices

🛡️ Security & Performance Assessment

• No Security Concerns: Changes are purely logical and don't introduce security vulnerabilities
• Performance Impact: Minimal - the _uri_tokens() function uses efficient regex operations and set operations
• Memory Usage: Creates small temporary sets during URI matching, which is acceptable

🔍 Technical Deep Dive

Core Fix Logic (field_detection.py:200-203)

def is_reference_uri(cls, uri: str) -> bool:
    uri_lower = uri.lower()
    if any(pattern in uri_lower for pattern in cls.MATCH_PATTERNS):
        return True
    return bool(_uri_tokens(uri) & set(cls.TOKEN_TERMS))

This elegantly handles both cases:

1. Specific patterns (Dublin Core, legacy) → substring matching (unchanged behavior)
2. Generic terms → token-based matching (new, fixed behavior)

Test Quality Analysis

The new test class TestReferenceURITokenBoundaries demonstrates excellent testing practices:

• Clear Intent: Class docstring explains the business logic being tested
• Comprehensive Coverage: Tests both the problem cases and preservation of existing functionality
• Maintainable Structure: Parametrized tests make it easy to add new test cases

🔧 Minor Observations

Mock Fixes (test_title_validation.py)

• Good Cleanup: Fixed incomplete mocks by adding missing slot_uri=None (lines 276, 277)
• Improved Robustness: These mocks now properly simulate real slot objects

Documentation Quality

• Excellent Examples: Docstrings include both positive and negative examples
• Clear Explanations: The PR description and code comments clearly explain the rationale

📋 Recommendations

1. ✅ Ready to Merge: This PR is well-implemented and ready for merging
2. Consider Follow-up: Future enhancement could add similar token-based logic to excerpt detection if needed
3. Documentation: The implementation serves as good documentation for future similar fixes

🎯 Conclusion

This is a high-quality PR that demonstrates:

• Problem-solving skills: Identified and fixed a subtle substring matching bug
• Engineering best practices: Test-first development, clear documentation, backward compatibility
• Code quality: Clean, maintainable implementation following project conventions

Recommendation: ✅ Approve and merge

The fix is targeted, well-tested, and maintains full backward compatibility while eliminating false positives that were causing spurious validation errors.

---