diff --git a/docs/reference/configuration/.pages b/docs/reference/configuration/.pages index d2fb347..3917c38 100644 --- a/docs/reference/configuration/.pages +++ b/docs/reference/configuration/.pages @@ -1,7 +1,10 @@ nav: - Application: application.md + - Cache: cache.md - Data sources: datasources.md - - File storage: file-storage.md - Database: database.md - Durable execution engine: dex-engine.md + - File storage: file-storage.md + - Task scheduler: task-scheduler.md + - Telemetry: telemetry.md - All properties: properties.md diff --git a/docs/reference/configuration/cache.md b/docs/reference/configuration/cache.md new file mode 100644 index 0000000..067dbff --- /dev/null +++ b/docs/reference/configuration/cache.md @@ -0,0 +1,77 @@ +# Cache + +Dependency-Track caches results of expensive operations to reduce load on +external services and speed up repeated lookups. Each cache has a TTL after +which entries expire and the cache provider drops them. + +Select the provider via [`dt.cache.provider`](properties.md#dtcacheprovider). + +## Providers + +### Database + +The `database` provider stores cache entries in the `CACHE_ENTRY` table on a +configured data source. This is the only built-in provider. + +!!! warning + The provider does not manage its own schema. The configured data source + must point at the same schema as the `default` data source, where Dependency-Track's + schema migrations create the `CACHE_ENTRY` table. + +!!! tip + Pointing the cache at a [secondary data source](datasources.md) that + targets the same schema can still be useful as a way to isolate cache + traffic from the main connection pool, so that cache activity doesn't + compete with the API server for database connections. + +A background maintenance worker periodically deletes expired rows and +refreshes the per-cache size counters that feed the cache metrics. + +Configuration: + +- [`dt.cache.provider.database.datasource.name`](properties.md#dtcacheproviderdatabasedatasourcename) +- [`dt.cache.provider.database.maintenance.initial-delay-ms`](properties.md#dtcacheproviderdatabasemaintenanceinitial-delay-ms) +- [`dt.cache.provider.database.maintenance.interval-ms`](properties.md#dtcacheproviderdatabasemaintenanceinterval-ms) + +## Entry TTLs + +Each named cache has its own TTL property of the form +`dt.cache."".ttl-ms`. + +### Vulnerability analyzer results + +Cached results of remote vulnerability analyzer lookups, keyed by component +identifier. Reduces API calls to upstream services across analyses of the same +components. + +Configuration: + +- [`dt.cache."vuln-analyzer.oss-index.results".ttl-ms`](properties.md#dtcachevuln-analyzeross-indexresultsttl-ms) +- [`dt.cache."vuln-analyzer.snyk.results".ttl-ms`](properties.md#dtcachevuln-analyzersnykresultsttl-ms) + +### Package metadata resolver responses + +Cached HTTP responses from package registry metadata endpoints, used to detect +outdated components. + +!!! tip + Entries include the response body together with `ETag` and `Last-Modified` validators + so that refreshes after the freshness window revalidate with a conditional request and + receive a 304 when the upstream response hasn't changed. The TTL must exceed the 12 h + freshness window for conditional requests to fire. + +Configuration: + +- [`dt.cache."package-metadata-resolver.cargo.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolvercargoresponsesttl-ms) +- [`dt.cache."package-metadata-resolver.composer.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolvercomposerresponsesttl-ms) +- [`dt.cache."package-metadata-resolver.cpan.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolvercpanresponsesttl-ms) +- [`dt.cache."package-metadata-resolver.gem.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolvergemresponsesttl-ms) +- [`dt.cache."package-metadata-resolver.github.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolvergithubresponsesttl-ms) +- [`dt.cache."package-metadata-resolver.gomodules.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolvergomodulesresponsesttl-ms) +- [`dt.cache."package-metadata-resolver.hackage.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolverhackageresponsesttl-ms) +- [`dt.cache."package-metadata-resolver.hex.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolverhexresponsesttl-ms) +- [`dt.cache."package-metadata-resolver.maven.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolvermavenresponsesttl-ms) +- [`dt.cache."package-metadata-resolver.nixpkgs.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolvernixpkgsresponsesttl-ms) +- [`dt.cache."package-metadata-resolver.npm.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolvernpmresponsesttl-ms) +- [`dt.cache."package-metadata-resolver.nuget.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolvernugetresponsesttl-ms) +- [`dt.cache."package-metadata-resolver.pypi.responses".ttl-ms`](properties.md#dtcachepackage-metadata-resolverpypiresponsesttl-ms) diff --git a/docs/reference/configuration/datasources.md b/docs/reference/configuration/datasources.md index 6f7d49d..f9998c6 100644 --- a/docs/reference/configuration/datasources.md +++ b/docs/reference/configuration/datasources.md @@ -1,4 +1,4 @@ -# Data Sources +# Data sources Data sources are logical objects through which database connections can be acquired. diff --git a/docs/reference/configuration/file-storage.md b/docs/reference/configuration/file-storage.md index 9f5f002..b97a615 100644 --- a/docs/reference/configuration/file-storage.md +++ b/docs/reference/configuration/file-storage.md @@ -1,4 +1,4 @@ -# File Storage +# File storage Dependency-Track uses file storage for intermediate data during background processing, including uploaded BOMs, vulnerability analysis results, and large notifications. diff --git a/docs/reference/configuration/task-scheduler.md b/docs/reference/configuration/task-scheduler.md new file mode 100644 index 0000000..cb363ff --- /dev/null +++ b/docs/reference/configuration/task-scheduler.md @@ -0,0 +1,73 @@ +# Task scheduler + +The task scheduler triggers recurring background activities like vulnerability +data source mirroring, integration uploads, metric updates, and maintenance +sweeps. Many tasks enqueue work for the [durable execution engine](dex-engine.md), +which executes workflows and other asynchronous processing. + +Nodes with the scheduler enabled coexist safely. The scheduler coordinates +execution through the database, so each due task runs on exactly one node +without requiring leader election. + +## Configuration + +You can turn the scheduler off on a per-node basis, for example to dedicate +specific nodes to serving web traffic. The `threads` setting caps how many +scheduled tasks may run concurrently on a single node. Because the +[durable execution engine](dex-engine.md) performs most asynchronous work, +this is rarely the right knob to tune. + +Configuration: + +- [`dt.task-scheduler.enabled`](properties.md#dttask-schedulerenabled) +- [`dt.task-scheduler.threads`](properties.md#dttask-schedulerthreads) +- [`dt.task-scheduler.poll-interval-ms`](properties.md#dttask-schedulerpoll-interval-ms) +- [`dt.task-scheduler.shutdown-max-wait-ms`](properties.md#dttask-schedulershutdown-max-wait-ms) + +## Cron expressions + +Each scheduled task takes its schedule from a `dt.task..cron` property, +which holds a five-field UNIX cron expression: + +```text +minute hour day-of-month month day-of-week +``` + +All expressions use **UTC**. + +Example, run every day at 03:30 UTC: + +```ini +dt.task.nvd-vuln-data-source-mirror.cron=30 3 * * * +``` + +## Scheduled tasks + +The scheduler ships with the following recurring tasks. Tasks marked with +[^1] also run once shortly after startup, with a random delay of up to one +minute. + +| Task | Property | Default cron | Purpose | +|-----------------------------------|-------------------------------------------------------------------------------------------------------------------------|------------------|-----------------------------------------------------------------------------------------------| +| Portfolio metrics update | [`dt.task.portfolio-metrics-update.cron`](properties.md#dttaskportfolio-metrics-updatecron) | `10 * * * *` | Refreshes per-project and portfolio time series metrics. | +| Vulnerability metrics update | [`dt.task.vuln-metrics-update.cron`](properties.md#dttaskvuln-metrics-updatecron) | `0 */6 * * *` | Refreshes counters used by the vulnerability dashboard. | +| Portfolio vulnerability analysis | [`dt.task.portfolio-analysis.cron`](properties.md#dttaskportfolio-analysiscron) | `0 6 * * *` | Re-analyzes every component in the portfolio against current vulnerability data. | +| NVD mirror[^1] | [`dt.task.nvd-vuln-data-source-mirror.cron`](properties.md#dttasknvd-vuln-data-source-mirrorcron) | `0 4 * * *` | Mirrors the NIST National Vulnerability Database. | +| GitHub Advisories mirror[^1] | [`dt.task.github-advisory-vuln-data-source-mirror.cron`](properties.md#dttaskgithub-advisory-vuln-data-source-mirrorcron) | `0 2 * * *` | Mirrors the GitHub Advisory Database. | +| OSV mirror[^1] | [`dt.task.osv-vuln-data-source-mirror.cron`](properties.md#dttaskosv-vuln-data-source-mirrorcron) | `0 3 * * *` | Mirrors the OSV vulnerability database. | +| EPSS mirror[^1] | [`dt.task.epss-mirror.cron`](properties.md#dttaskepss-mirrorcron) | `0 1 * * *` | Mirrors the FIRST EPSS scores feed. | +| Vulnerability database maintenance | [`dt.task.vuln-database-maintenance.cron`](properties.md#dttaskvuln-database-maintenancecron) | `0 0 * * *` | Removes orphaned vulnerability records and reconciles indexes. | +| Vulnerability policy bundle sync[^1] | [`dt.task.vuln-policy-bundle-sync.cron`](properties.md#dttaskvuln-policy-bundle-synccron) | `*/15 * * * *` | Pulls the configured vulnerability policy bundle. No-op when `dt.vuln-policy-bundle.url` is empty. | +| Package metadata resolution | [`dt.task.package-metadata-resolution.cron`](properties.md#dttaskpackage-metadata-resolutioncron) | `0 1 * * *` | Refreshes outdated-component status for portfolio components. | +| Package metadata maintenance | [`dt.task.package-metadata-maintenance.cron`](properties.md#dttaskpackage-metadata-maintenancecron) | `0 */12 * * *` | Removes stale and orphaned package metadata records. | +| Metrics maintenance | [`dt.task.metrics-maintenance.cron`](properties.md#dttaskmetrics-maintenancecron) | `1 * * * *` | Compacts time series metric history. | +| Tag maintenance | [`dt.task.tag-maintenance.cron`](properties.md#dttasktag-maintenancecron) | `0 */12 * * *` | Removes unused tags. | +| Project maintenance | [`dt.task.project-maintenance.cron`](properties.md#dttaskproject-maintenancecron) | `0 */4 * * *` | Applies project retention and cleanup rules. | +| Fortify SSC upload | [`dt.task.fortify-ssc-upload.cron`](properties.md#dttaskfortify-ssc-uploadcron) | `0 2 * * *` | Uploads finding reports to Fortify SSC. | +| DefectDojo upload | [`dt.task.defect-dojo-upload.cron`](properties.md#dttaskdefect-dojo-uploadcron) | `0 2 * * *` | Uploads finding reports to DefectDojo. | +| Kenna Security upload | [`dt.task.kenna-security-upload.cron`](properties.md#dttaskkenna-security-uploadcron) | `0 2 * * *` | Uploads finding reports to Kenna Security. | +| Expired session cleanup | [`dt.task.expired-session-cleanup.cron`](properties.md#dttaskexpired-session-cleanupcron) | `0 * * * *` | Deletes expired user session tokens. | +| Scheduled notification dispatch | [`dt.task.scheduled-notification-dispatch.cron`](properties.md#dttaskscheduled-notification-dispatchcron) | `* * * * *` | Polls for due scheduled notification rules and dispatches them. | +| Telemetry submission[^1] | [`dt.task.telemetry-submission.cron`](properties.md#dttasktelemetry-submissioncron) | `0 */1 * * *` | Submits anonymous usage data. See [Telemetry](telemetry.md). | + +[^1]: Triggered once on startup with a random delay of up to one minute, then on the configured schedule. diff --git a/docs/reference/configuration/telemetry.md b/docs/reference/configuration/telemetry.md new file mode 100644 index 0000000..54da33e --- /dev/null +++ b/docs/reference/configuration/telemetry.md @@ -0,0 +1,78 @@ +# Telemetry + +Dependency-Track periodically submits a small, anonymous data point to +`https://metrics.dependencytrack.org`. This helps the maintainers understand +adoption, plan version support, and decide which database systems to keep +testing against. + +The submission does not include information that could tie data back to a +specific organization, such as IP addresses, hostnames, or project content. + +## Data collected + +Each submission is a single JSON object with four fields: + +| Field | Description | +|--------------|-----------------------------------------------------------------------------| +| `system_id` | Random cluster identifier generated at first startup. Stable across restarts and node replacements. | +| `dt_version` | Dependency-Track release version. | +| `db_type` | Database product name reported by JDBC (for example, `PostgreSQL`). | +| `db_version` | Database product version reported by JDBC. | + +The `system_id` lets the maintainers correlate submissions from the same +deployment over time without identifying who runs it. + +## Submission schedule + +The [task scheduler](task-scheduler.md) runs the telemetry submission task +once shortly after startup with a random delay of up to one minute, and then +on the schedule defined by +[`dt.task.telemetry-submission.cron`](properties.md#dttasktelemetry-submissioncron) +(default `0 */1 * * *`, hourly). + +The task itself enforces a 24-hour interval between submissions, so the cron +expression controls how often the task checks whether a new submission is +due, not how often data is actually sent. + +## Opting out + +You can opt out at any time through any of the following methods. + +### Administration UI + +Navigate to **Administration** > **Configuration** > **Telemetry** and turn +off submission. + +### Configuration API + +```shell linenums="1" +curl -X POST \ + -H 'X-Api-Key: odt_...' \ + -H 'Content-Type: application/json' \ + -d '{ + "groupName": "telemetry", + "propertyName": "submission.enabled", + "propertyValue": "false" + }' \ + https://dtrack.example.com/api/v1/configProperty +``` + +### Before first startup + +To opt out before the first startup populates the runtime setting, set +[`dt.telemetry.submission.default-enabled`](properties.md#dttelemetrysubmissiondefault-enabled) +to `false`: + +```ini +dt.telemetry.submission.default-enabled=false +``` + +As an environment variable: + +```shell +DT_TELEMETRY_SUBMISSION_DEFAULT_ENABLED=false +``` + +This property only seeds the initial value of the runtime setting. Once the +setting exists in the database, changes via the UI or REST API override the +initial value. diff --git a/mkdocs.yml b/mkdocs.yml index fa97cd8..58a2d87 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -81,6 +81,7 @@ markdown_extensions: - abbr - admonition - attr_list +- footnotes - md_in_html - toc: permalink: true