Skip to content

data: add offline library catalog database (v1, 188,512 records) - #24

Closed
U-Illll wants to merge 5 commits into
wormforce:mainfrom
U-Illll:add-library-catalog-data
Closed

U-Illll wants to merge 5 commits into
wormforce:mainfrom
U-Illll:add-library-catalog-data

Conversation

@U-Illll

@U-Illll U-Illll commented Sep 18, 2026

Copy link
Copy Markdown

What

Adds an offline library catalog database for library search / library detail,
compiled from the public Primo VE catalog export (2026-09-18, 188,512 records).
No code changes in this PR.

Contents (data/library-catalog/)

File Description
cards.jsonl.gz 188,512 bibliographic cards (mms/title/creator/publisher/year/language/type/holdings/location/CLC class)
embeddings_int8.bin (+size scale, PCA components) bge-small-zh-v1.5 vectors, 512-d projected to 384-d, int8-quantized
knn_indptr/indices/sims semantic link graph, top-10 neighbors per card (CSR)
clusters.i32.bin + clusters.json 400 topic clusters with keyword directory
tfidf/ sparse keyword index (338,358 terms, CSR + vocab + idf)
subjects.json / shelves.json CLC subject classes and shelf-level profiles
manifest.json per-file sha256, format notes, Primo docId mapping
examples/node-client.js zero-dependency reference client (search / similar / links / cluster / docid)
README.md file formats + semantic-search integration pipeline

Size note

Total ~131 MB (largest file 72.4 MB). Happy to convert to Git LFS, or trim to a
smaller core subset, if you prefer — let me know.

Primo mapping

  • docId = L:alma<mms>; official detailUrl template in manifest.json
  • creators <- creator.split(";"); date <- year; format <- type

Source

Public bibliographic facts from the library's public read-only Primo interface
(no login required).

@pentaoa

pentaoa commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

你好你好,Thank you for putting this together — the amount of work behind collecting, cleaning, indexing, quantizing, and documenting 188,512 catalog records is genuinely impressive. The manifest and reference client also make the dataset much easier to understand.

After reviewing the proposal, we don’t think the dataset is a good fit for the main sustech-cli repository in its current form:

  • sustech-cli already uses Primo’s public JSON search and detail APIs, so normal catalog queries can retrieve current data online without bundling a snapshot.
  • This PR contains data only and does not integrate the dataset into the existing CLI commands.
  • The roughly 131 MB of generated data would permanently increase the Git repository size, while data/ is not currently included in the published npm package.
  • As a point-in-time catalog snapshot, it would need a reproducible refresh and distribution process to avoid becoming stale.

This is not a negative assessment of the dataset itself. The offline TF-IDF index, embeddings, similarity graph, and clusters could be valuable for offline or semantic-search use cases that Primo does not provide.

A better architecture would be to publish the dataset separately—such as in a dedicated repository, a versioned GitHub Release asset, or a standalone data package—and let the CLI download it optionally in the future. That would preserve the work while keeping the core repository and default installation lightweight.

For those reasons, we’re going to close this PR without merging it. We’d be very happy to discuss a smaller follow-up proposal for optional dataset downloading/caching, or improvements that build on the existing live Primo integration.

Thank you again for the thoughtful contribution and for documenting it so thoroughly. 🤗

@pentaoa pentaoa closed this Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants