Repository navigation
feat: collector that builds, signs and verifies public program bundles - #1
Merged
Merged
Conversation
Collects public bug-bounty and vulnerability-disclosure programs from open, machine-readable sources, normalises them into one record format with typed scope, validates every record and scope entry, and publishes a signed snapshot and delta per run. - sources: disclose.io diodb (CC0-1.0) and RFC 9116 security.txt over the CISA dotgov-data federal domain list (CC0-1.0) plus a repository seed file - polite fetch client: named User-Agent and contact, robots.txt, one request per second per host, conditional requests, size caps, public addresses only - scope rules: public suffixes, reserved names, private/reserved ranges and CIDRs wider than /16 or /48 are rejected with the reason; out of scope wins - change log per run: programs added/closed/reopened/dropped, targets added/removed/modified, rules, terms and status changes - bundle and trust model shared with the vulnerability feed: DSSE envelopes, offline root and versioned key set, monotonic sequence, 7-day expiry - daily publish workflow (build job without the key, publish job in the publish environment); JSON Schemas; recorded fixtures, no network in tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The first version of the public program collector: it reads open, machine-readable sources of public bug-bounty and vulnerability-disclosure programs, normalises them (typed scope, rules, terms, contact, provenance), validates every record and scope entry, and publishes a signed snapshot plus delta per run as a GitHub Release. The OpenCTEM platform imports the bundle into its public program monitor.
The repository layout, signing (DSSE, offline root, versioned key set, monotonic sequence, 7-day expiry), caps and verification checks follow the OpenCTEM vulnerability feed, so the platform can share one verifier; only the payload types (
application/vnd.openctem.programfeed.*) and record files (programs,changes) differ.Sources
seeds/securitytxt.txtPolicyfield (5 open, 6 paused because the file expired)Live dry run (2026-10-10, not published): 2,435 programs (2,408 open, 27 paused, 440 bounty), 67 inferred targets, 0 rejected. A second run with
--prevgave 0 changes (stable ids and ordering), and replaying the same sequence was refused.Threat model
publishenvironment; sequence and expiry checks). A key set or manifest of another feed is refused (feed-specific payload types, test included).*.com,*.co.uk,*.github.io), reserved names, private/reserved/documentation/metadata addresses and ranges, CIDRs wider than /16 or /48, credentialed or non-http(s) URLs and control/bidi characters are moved torejectedwith the reason. The verifier refuses a whole bundle carrying one such target even under a valid signature (tests).confidence: inferred, andscope_publishedsays whether the source has structured scope at all. Nothing in this feed authorises testing.From: security@openctem.io, robots.txt honoured (an unreachable robots.txt means do not fetch), 1 request per second per host, conditional requests from a cache kept between runs. No login, cookies or bot-protection bypass.Owner actions
keys/README.md), with a root distinct from the vulnerability feed root: commitkeys/root-keyid.txtandkeys/keyset.dsse.json.publish(deployments frommainonly) with the secretPROGRAMFEED_SIGNING_KEY. No other secret is needed.PROGRAMFEED_ROOT_KEY_ID).Until step 1 is done the scheduled workflow logs a warning and publishes nothing.
Verification
gofmt,go vet,go test -race ./...(recorded fixtures, no network), 30 s fuzz of the scope normaliser, govulncheck clean. End to end on live data: keys, build, sign, verify, build with--prev, sign, verify; replay refused.