chore(embeddings): upgrade Transformers.js to v4

[aolin480]

Summary

• Upgrade @huggingface/transformers from ^3.8.1 to the current stable ^4.2.0.
• Upgrade the pinned native runtime from onnxruntime-node@1.24.2 to the v4 dependency version, 1.24.3.
• Replace v3-only/root type usage with v4's exported FeatureExtractionPipeline and PreTrainedTokenizer types.
• Preserve the reranker tokenizer-pair fix by continuing to pass passages through text_pair.
• Keep the release-truth and macOS path-canonicalization fixes already included on this branch.

Closes #72.

Why

Issue #72 was opened while Transformers.js v4 was still prerelease. v4 is now stable on npm, and the current release uses the ONNX Runtime family that contains the macOS mutex-destruction fix tracked by #68.

The package's pipeline/model-loading APIs used here remain behaviorally compatible. The breaking change in this codebase was limited to exported TypeScript type names.

macOS verification

Tested on macOS 26.5.2 (arm64), Node v26.5.0:

• Fresh 34.7 MB model cache created successfully.
• Local feature-extraction pipeline returned a finite 384-dimensional embedding.
• Cross-encoder reranker loaded, scored three passages, and reported active.
• Three separate Node processes generated embeddings and exited cleanly.
• The prior mutex lock failed: Invalid argument hard-crash did not reproduce.

This is evidence for the tested Apple Silicon environment; it is not a claim that every macOS/Node combination has been exhaustively covered.

Verification

• pnpm run type-check
• pnpm run build
• pnpm run format:check
• pnpm test — 602 tests passed
• Focused reranker/embedding tests — 17 tests passed
• Fresh-cache native Hugging Face v4 embedding + reranker smoke
• Three-process macOS startup/embedding/clean-exit smoke

[gemini-code-assist]

Code Review

This pull request updates the CrossEncoderTokenizer interface and its usage in the reranker to align with Hugging Face Transformers tokenizer options, refactors release tests to dynamically use the package version instead of hardcoded strings, and introduces path normalization in contextbench tests to ensure consistent path comparisons. Feedback suggests correcting return_tensor to return_tensors in the tokenizer interface and safely handling an optional options parameter in the test mock to avoid potential runtime errors.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[greptile-apps]

Greptile Summary

This PR fixes the tokenizer call shape in the cross-encoder reranker to use the text_pair named option rather than a positional second argument, aligning with the @huggingface/transformers v3 API. It also makes version-tracking tests forward-compatible (no more hardcoded 2.2.0) and applies realpathSync to resolve macOS /tmp → /private/tmp symlink mismatches in the ContextBench path assertions.

• src/core/reranker.ts: scorePair now calls cachedTokenizer(query, { text_pair: passage, padding, truncation, max_length }) instead of the three-positional-argument form; the CrossEncoderTokenizer interface is updated accordingly.
• tests/reranker-recovery.test.ts: Mock tokenizer and the new toHaveBeenCalledWith assertion both reflect the corrected call shape.
• tests/release-truth-surfaces.test.ts / .github/workflows/publish-npm-on-release.yml: Version references are now derived from package.json rather than hardcoded to a specific release, so the tests stay green across future bumps.

Confidence Score: 4/5

The change is safe to merge — the core tokenizer call shape fix is correct and the mock/assertion in the test correctly reflects the new API surface.

The reranker call shape change is straightforward and well-tested. The only notable item is a typo in the newly added CrossEncoderTokenizer interface (return_tensor instead of return_tensors), which has no runtime impact today since that field is never passed, but could silently mislead a future caller.

The CrossEncoderTokenizer interface in src/core/reranker.ts has the minor field-name discrepancy worth fixing before it propagates.

Important Files Changed

Filename	Overview
src/core/reranker.ts	Core reranker fix: tokenizer call shape updated from positional to text_pair named option; interface expanded to match transformers v3 API, though return_tensor (singular) in the new interface appears to be a typo for return_tensors.
tests/reranker-recovery.test.ts	Mock tokenizer updated to match new single-text + options call shape; a new toHaveBeenCalledWith assertion validates the text_pair field but only for one of the three scored results.
tests/release-truth-surfaces.test.ts	Version checks now derive expectedVersion from package.json rather than hardcoding 2.2.0; test descriptions updated to match; safe and correct change.
tests/contextbench-baseline-schema-gate.test.ts	Adds normalizeFilesystemPath using realpathSync to resolve macOS symlinks before comparing cwd capture paths; applied consistently at three assertion sites.
src/embeddings/transformers.ts	Comment-only change: updates inline TSC workaround note for clarity; no runtime behavior affected.
.github/workflows/publish-npm-on-release.yml	Manual dispatch default tag updated from v2.2.0 to v2.3.0 to match current release; aligns with the now-dynamic release truth test.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Caller
    participant rerank
    participant ensureModelLoaded
    participant AutoTokenizer
    participant AutoModel
    participant scorePair

    Caller->>rerank: rerank(query, results)
    rerank->>rerank: isAmbiguous(results)?
    alt scores not ambiguous
        rerank-->>Caller: return results (unchanged)
    else scores ambiguous
        rerank->>ensureModelLoaded: ensureModelLoaded()
        ensureModelLoaded->>AutoTokenizer: from_pretrained(model)
        AutoTokenizer-->>ensureModelLoaded: cachedTokenizer
        ensureModelLoaded->>AutoModel: "from_pretrained(model, {dtype:'q8'})"
        alt model load fails (corrupt cache)
            AutoModel-->>ensureModelLoaded: Error (Protobuf/parse)
            ensureModelLoaded->>ensureModelLoaded: resetLoadedState()
            ensureModelLoaded->>ensureModelLoaded: rmSync(modelCacheDir)
            ensureModelLoaded-->>rerank: throw Error
            rerank-->>Caller: throw Error
            Note over Caller: Next call retries from scratch
        else model load succeeds
            AutoModel-->>ensureModelLoaded: cachedModel
            ensureModelLoaded-->>rerank: resolved
            loop for each result in top-K
                rerank->>scorePair: scorePair(query, passage)
                scorePair->>scorePair: "cachedTokenizer(query, { text_pair: passage, padding, truncation, max_length: 512 })"
                scorePair->>cachedModel: cachedModel(inputs)
                cachedModel-->>scorePair: "{ logits: { data } }"
                scorePair-->>rerank: logits.data[0]
            end
            rerank->>rerank: sort by crossScore desc
            rerank->>rerank: apply sigmoid to scores
            rerank-->>Caller: [...reranked, ...rest]
        end
    end

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Caller
    participant rerank
    participant ensureModelLoaded
    participant AutoTokenizer
    participant AutoModel
    participant scorePair

    Caller->>rerank: rerank(query, results)
    rerank->>rerank: isAmbiguous(results)?
    alt scores not ambiguous
        rerank-->>Caller: return results (unchanged)
    else scores ambiguous
        rerank->>ensureModelLoaded: ensureModelLoaded()
        ensureModelLoaded->>AutoTokenizer: from_pretrained(model)
        AutoTokenizer-->>ensureModelLoaded: cachedTokenizer
        ensureModelLoaded->>AutoModel: "from_pretrained(model, {dtype:'q8'})"
        alt model load fails (corrupt cache)
            AutoModel-->>ensureModelLoaded: Error (Protobuf/parse)
            ensureModelLoaded->>ensureModelLoaded: resetLoadedState()
            ensureModelLoaded->>ensureModelLoaded: rmSync(modelCacheDir)
            ensureModelLoaded-->>rerank: throw Error
            rerank-->>Caller: throw Error
            Note over Caller: Next call retries from scratch
        else model load succeeds
            AutoModel-->>ensureModelLoaded: cachedModel
            ensureModelLoaded-->>rerank: resolved
            loop for each result in top-K
                rerank->>scorePair: scorePair(query, passage)
                scorePair->>scorePair: "cachedTokenizer(query, { text_pair: passage, padding, truncation, max_length: 512 })"
                scorePair->>cachedModel: cachedModel(inputs)
                cachedModel-->>scorePair: "{ logits: { data } }"
                scorePair-->>rerank: logits.data[0]
            end
            rerank->>rerank: sort by crossScore desc
            rerank->>rerank: apply sigmoid to scores
            rerank-->>Caller: [...reranked, ...rest]
        end
    end

Loading

_{Reviews (1): Last reviewed commit: "test(release): Keep current version chec..." | Re-trigger Greptile}