Skip to content

Add Claude skill - #169

Open
JojiiOfficial wants to merge 8 commits into
devfrom
agents-skill
Open

Add Claude skill#169
JojiiOfficial wants to merge 8 commits into
devfrom
agents-skill

Conversation

@JojiiOfficial

@JojiiOfficial JojiiOfficial commented Aug 21, 2026

Copy link
Copy Markdown
Member

Adds a skill for claude code to run bfb with all context it needs.
It covers all commands (upload, search, scroll, schema), experiment templates, how to download real data and a list of common mistakes.

Validation

Additionally, this PR adds a new subcommand validate, which I found agents were frequently looking for while testing the skill.
It checks each config parses and passes its own validation, then cross-references the upload against the search/scroll configs: vector names, kinds and dimensions, and filter fields against declared indexes.
Obvious things that Qdrant will always reject (e.g. dim mismatch) will lead to an error. Likely misconfigurations, such as random vectors in search but real vectors in the upload config will throw a warning.

Tests

Two agents were given identical bfb tasks: one with the skill available, one without (control).
Note that these numbers fully depend on the initial prompt. The numbers here represent a selected set of smoke tests to verify the skill helps and how much each metric shifts for the specific test prompt.
These numbers don't represent real usage speedups.

Averages across all 5 runs

Metric With skill Control (no skill) Delta
Tool calls 12.6 30.8 2.4× fewer
Wall time 2m53s 5m33s 1.9× faster
Tokens 58.0k 79.9k 1.4× fewer
Recall@10 (dbpedia) 0.612 0.612 identical

bfb source-code accesses (counted only in 2 of the 5 runs)

Task With skill Control
Upload-config authoring 0 2 files + 3 greps
Search benchmark 0 0

Per-run detail

Run Task Calls (Skill) Calls (no skill) Skill time Ctrl time
1 upload+search (no server given) 27 45 4m55s 8m45s
2 read-only search 11 39 2m33s 5m09s
3 read-only search 10 31 3m12s 5m45s
4 upload-config authoring (offline) 6 18 1m21s 3m55s
5 read-only search 9 21 2m23s 4m10s

Stale Readme section

This PR also removes a stale section about shell completions from the README.md. We currently have it specified twice and my changes remove the old version.

@JojiiOfficial
JojiiOfficial marked this pull request as draft August 21, 2026 07:48
@JojiiOfficial
JojiiOfficial marked this pull request as ready for review September 7, 2026 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant