Repository navigation
feat: dataset adapters, per-platform publish policy, takedown and outreach drafts - #2
Merged
Merged
Conversation
added 2 commits
October 10, 2026 05:03
Collects public bug-bounty and vulnerability-disclosure programs from open, machine-readable sources, normalises them into one record format with typed scope, validates every record and scope entry, and publishes a signed snapshot and delta per run. - sources: disclose.io diodb (CC0-1.0) and RFC 9116 security.txt over the CISA dotgov-data federal domain list (CC0-1.0) plus a repository seed file - polite fetch client: named User-Agent and contact, robots.txt, one request per second per host, conditional requests, size caps, public addresses only - scope rules: public suffixes, reserved names, private/reserved ranges and CIDRs wider than /16 or /48 are rejected with the reason; out of scope wins - change log per run: programs added/closed/reopened/dropped, targets added/removed/modified, rules, terms and status changes - bundle and trust model shared with the vulnerability feed: DSSE envelopes, offline root and versioned key set, monotonic sequence, 7-day expiry - daily publish workflow (build job without the key, publish job in the publish environment); JSON Schemas; recorded fixtures, no network in tests
…, outreach drafts Adds adapters for two public aggregate datasets of bug-bounty program listings (read at one resolved commit, provenance per record, targets marked published_by_platform), a per-platform publish policy so only allowed platforms reach the signed feed, a local-only unsigned bundle mode, TAKEDOWN.md and unsent permission-request drafts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge after #1 (stacked on feat/initial-collector; the diff against main includes #1 until it merges).
What
bounty-targets(arkadiyt/bounty-targets-data: HackerOne, Bugcrowd, Intigriti, YesWeHack) andpd-bounty(projectdiscovery/public-bugbounty-programs). Files are read at one commit resolved per run (raw URLs contain the commit; the resolve call uses ETag caching), size and record caps, strict parsing; the program page must sit on the platform host or the record is skipped.published_by_platform; provenance gainsdataset,dataset_commit,original_platform,original_url(all four together or none).policy/platforms.json: per platformpublish(default false for everything except self-hosted),attribution,takedown_contact,reason; unlisted = not published;excluded_programsfor program-level takedown. Applied to the finished snapshot and to the previous release (switching a platform off emitsprogram_dropped). A signed build does not fetch unpublished dataset platforms.build --local-only: ignores the policy, unsigned bundle markedcollector.local_only;signrefuses it.Threat model
published_by_platformis never permission to test.Tests
Recorded fixtures only (no network):
go test ./...green. Also ran the real binary on pd-bounty (802 programs local-only, 226 signed-mode).Follow-up
The OpenCTEM importer must accept
published_by_platformand the new provenance fields before a bundle with them is published; with the default policy no such record is published yet.