Skip to content

Fix/finding api vulnerability id prefetch - #16658

Closed
jensholdgaard wants to merge 26 commits into
DefectDojo:masterfrom
jensholdgaard:fix/finding-api-vulnerability-id-prefetch
Closed

jensholdgaard wants to merge 26 commits into
DefectDojo:masterfrom
jensholdgaard:fix/finding-api-vulnerability-id-prefetch

Conversation

@jensholdgaard

Copy link
Copy Markdown

⚠️ Pre-Approval check ⚠️

We don't want to waste your time, so if you're unsure whether your hypothetical enhancement meets the criteria for approval, please file an issue to get pre-approval before beginning work on a PR.
Learn more here: https://github.com/DefectDojo/django-DefectDojo/blob/master/readme-docs/CONTRIBUTING.md#submission-pre-approval

Description

Describe the feature / bug fix implemented by this PR.
If this is a new parser, the parser guide may be worth (re)reading.

Test results

Ideally you extend the test suite in tests/ and dojo/unittests to cover the changed in this PR.
Alternatively, describe what you have and haven't tested.

Documentation

Please update any documentation when needed in the documentation folder)

Checklist

This checklist is for your information.

  • Make sure to rebase your PR against the very latest dev.
  • Submit all PRs, features and bug fixes alike, against the dev branch.
  • Give a meaningful name to your PR, as it may end up being used in the release notes.
  • Your code is Ruff compliant (see ruff.toml).
  • Your code is python 3.13 compliant.
  • If this is a new feature and not a bug fix, you've included the proper documentation in the docs at https://github.com/DefectDojo/django-DefectDojo/tree/dev/docs as part of this PR.
  • Model changes must include the necessary migrations in the dojo/db_migrations folder.
  • Add applicable tests to the unit tests.
  • Add the proper label to categorize your PR.

Extra information

Please clear everything below when submitting your pull request, it's here purely for your information.

Moderators: Labels currently accepted for PRs:

  • Import Scans (for new scanners/importers)
  • enhancement
  • performance
  • feature
  • bugfix
  • maintenance (a.k.a chores)
  • dependencies
  • New Migration (when the PR introduces a DB migration)
  • settings_changes (when the PR introduces changes or new settings in settings.dist.py)

Contributors: Git Tips

Rebase on dev branch

If the dev branch has changed since you started working on it, please rebase your work after the current dev.

On your working branch mybranch:

git rebase dev mybranch

In case of conflict:

 git mergetool
 git rebase --continue

When everything's fine on your local branch, force push to your myOrigin remote:

git push myOrigin --force-with-lease

To cancel everything:

git rebase --abort

Squashing commits

git rebase -i origin/dev
  • Replace pick by fixup on the commits you want squashed out
  • Replace pick by reword on the first commit if you want to change the commit message
  • Save the file and quit your editor

Force push to your myOrigin remote:

git push myOrigin --force-with-lease

DefectDojo release bot and others added 26 commits October 30, 2026 02:27
…ter-into-dev-3.4.304

Release: Merge back 3.4.304 into dev from: release/merge-master-into-dev-3.4.304
…tDojo#16547)

* feat(locations): LocationData builders for domain, ip and cert

Parsers can report a DNS name, an IP address or an X.509 certificate as a
typed location: LocationData.domain(), LocationData.ip() and
LocationData.cert(). The builders emit a fixed set of keys so every consumer
reads one shape; normalizing the values is the consumer's job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(locations): read a single SAN as one name and document the cert and ip rules

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#16637)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…jo#16632)

* perf(qualys): look up QID details once per report, not once per finding

The Qualys XML parser found each finding's QID details with a search from
the document root, which walks every host before it reaches the glossary.
Parsing cost findings x report size: a 56-host, 2,408-finding report took
40.0 s of CPU to parse, against 2.0 s for 7 hosts.

The glossary is now indexed once per report and each finding reads its
entry from that index. The first entry for an id still wins, as the search
returned it, and the parsed findings are identical to before on reports of
7 to 56 hosts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(qualys): count per-QID lookups wherever the search starts

The guard only watched searches made on the whole tree, so a lookup made
from the root element once per finding would have brought the quadratic
parse back without failing it. Every find, findall, findtext and iterfind,
on a tree or an element, resolves its path through ElementPath.iterfind,
so that is what the test counts now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…nd records screenshots (DefectDojo#16639)

- API v2 Sample Code: query /api/v2/findings/ (readable with any token) instead of
  /api/v2/users/, print the results instead of the Response internals, show the
  paginated response shape and a filtered query, with placeholder host and token.
- do_not_reactivate: a reimport keeps a closed finding closed; new and open
  findings are processed as usual.
- Location page: its table lists open findings only; say how to reach every
  finding on a Location through the Total Findings count.
- Location Map: link it from the Locations overview, and state what a Reader can
  and cannot change.
- Recapture the upstream records screenshots with the row menu.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…le page (DefectDojo#16614)

* docs(navigation): describe the Preferences page and its landing page

Add a Preferences page for DefectDojo Pro: the default landing page
applied after sign-in, the default dashboard, saving, and the feature
flag. The sidebar page names Preferences in the account menu, and the
custom dashboards page points at it for changing the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(navigation): call the fallback page Home and cover identity-provider sign-ins

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(navigation): the default landing page is set on the User Profile page

The landing page moves from a Preferences page to a Signing In field on
User Profile, and the default dashboard stays on the Dashboards page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e tiles (DefectDojo#16640)

The Asset page's Finding Distribution panel is now a ring of Active,
Mitigated and Other findings, whose legend links to each status, beside
the static and dynamic shares. English page only; the language copies
are translated automatically.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…es how common a tag is (DefectDojo#16634)

* perf(findings): resolve tag filter names to tag ids before matching

tag_exists() matched the tag through a join on the tag table's name, so the
planner could not tell which tag a query was about and assumed the average
number of rows per tag. For a tag carried by most findings that estimate was
off by three orders of magnitude (573 expected, 400,001 actual), and the
finding list's COUNT and id queries took a nested-loop semi-join that probed
every tagged row: 20.6 s for one COUNT over 400k tagged findings, 2.5 s once
the subquery names the tag id.

The matching tags are now read first (one indexed lookup on the tag table)
and the EXISTS subquery tests the through table against those ids. A lookup
that matches more than TAG_ID_LITERAL_LIMIT tags (a short icontains value)
keeps the join. Results, counts and paging are unchanged: the existing
equivalence tests against the old JOIN form pass as they are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(findings): resolve every AND-ed tag in one read, and match nothing for an unknown one

tags__and (and its test__, test__engagement__ and test__engagement__product__
forms) looked each listed tag up on its own, one query per tag. The names
are now resolved in one name__in read, and one EXISTS per tag id is built
from it. A listed name that is no tag cannot be present, so the filter
returns an empty queryset without building the subqueries.

Tests: the over-limit fallback is checked on the rows it returns, not only
its SQL, and an unknown tag is checked through the FilterSet for tags__and,
test__engagement__tags__and and not_tags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…DefectDojo#16642)

Describe the enterprise option to run Sensei on a Cloud instance's own model
provider account through AI Model Settings, how it differs from on-premise
(keys required, public base URL only, no fallback to DefectDojo's model), and
point readers to hello@defectdojo.com to request it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(models): add a default-queryset hook for plugins

Adds dojo.query_hooks: a registry of default-queryset filters keyed by model,
applied by the HookedManager declared as `objects` on Product_Type, Product,
Engagement, Test, Finding and the rows that hang off them. With nothing
registered the querysets are unchanged. Each of those models also gets an
explicit `unfiltered` manager, and two nullable columns (archived_at,
archive_batch) that open source never reads or writes, with partial indexes
built concurrently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(models): renumber the queryset-hook migrations after dev

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: expect the archive columns on endpoint status rows

test_endpoint_queries compares Endpoint_Status.values() against literal
rows, which now include the two nullable columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(migrations): hold the archive-column locks only for the SQL itself

Measured on 10 million findings, 0304 blocked every read of dojo_finding for
about 40 seconds. Its SQL takes milliseconds, but Django rebuilds its model
state after each of the migration's operations, and each table's
ACCESS EXCLUSIVE lock is taken by its ALTER and held through all the
rebuilding that follows until the migration commits.

The operations are now applied to the migration state only, and the database
work is the SQL they generate (from sqlmigrate), frozen and run as one batch at
the end. The resulting schema is identical; the slowest concurrent read during
the migration dropped to 0.4 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(dedupe): let a plugin set excess duplicates aside instead of deleting them

register_excess_duplicate_handler installs one handler for the chunks the
excess-duplicate task removes. It runs inside the chunk's transaction with
the rows locked; when it returns True nothing is deleted. With no handler,
or one that returns False, the chunk is deleted as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…jo#16563)

* docs(asset-modelling): document Archive and retention rules

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(asset-modelling): say archived data still counts toward the license

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(archive): archiving and restoring run in the background

The object disappears at once and the rest of its tree is archived in the
background; restores work the same way. A bulk selection or one Triage Engine
run is one archive, restored as a whole. Archive and unarchive return 202.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(archive): say what stays visible while archiving, and the parent rule on restore

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(archive): archived data does not count against the license; excess duplicates are archived

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(archive): archiving works under a breached license, restoring does not

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…api/v2/findings/ (DefectDojo#16644)

* fix(findings-api): honor ?engagement=, ?product= and ?prod_type= on /api/v2/findings/

A Finding has no engagement, product or prod_type field and ApiFindingFilter declared
no filter by those names, so django-filter dropped the parameters and the list answered
with every finding the caller could see. Declare them as short names for the
test__engagement, test__engagement__product and test__engagement__product__prod_type
filters, and document them on the Findings pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(findings-api): pin that the short names do not widen for a product member; note the change in the 3.5 upgrade page

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…16647)

The built-in API Importer role holds no Note permissions, so the chart
should not show it adding, editing or deleting notes.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…plates and the mandatory-field error (DefectDojo#16574)

* docs(connectors): document ServiceDesk Plus custom fields, ticket templates and the mandatory-field error

* docs(connectors): ServiceDesk Plus Ticket Template sets the subject, and fallbacks are listed under Errors
…fectDojo#16626)

Maps a comma-separated DD_SECRET_KEY_FALLBACKS to Django's
SECRET_KEY_FALLBACKS, so an operator can change DD_SECRET_KEY and keep
values encrypted under the old key readable. The restoring page now says
the SSO, OIDC, LDAP and email secrets use DD_SECRET_KEY too, and how to
change the key.
…ojo#16645)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* docs(connectors): Security Hub ECR images in use only

Documents the ECR Images in Use Only toggle of the Security Hub
connector, its Amazon Inspector permission and its limits. Names the
form section Import Configuration, as the form shows it.

* docs(connectors): Security Hub ECR in-use Sync cost and scan on push

* docs(connectors): Security Hub ECR in-use limits, save check and skip scope

* docs(connectors): Security Hub ECR in-use source link, scanning per region and multi-arch images

* docs(connectors): Security Hub ECR in-use reads Inspector at least once per Record

* docs(connectors): Security Hub ECR in-use findings can stay open up to two days after an image stops

* docs(connectors): Security Hub records Inspector cannot see now fail instead of skipping

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…fectDojo#16651)

* docs(onprem): document the web-request statement_timeout and its settings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(onprem): statement timeout on Kubernetes, poolers and the capped long bound

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(onprem): long statement timeout default when harakiri is unset

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…Dojo#16578)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…jo#16650)

Login Settings gains Idle Timeout, and Session Age is now an absolute
lifetime counted from the login. Adds a Session expiry section and corrects
who can change the page (the Tuner Edit permission).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ojo#16654)

The page layouts and EU CRA pages said the Organization Memberships,
Asset Identity, Exposure and Regulatory cards sit at the top of the
Asset page and that the shipped layout places them first. The shipped
layout now starts with the Open Finding Severity meter and places the
cards after the Details row, memberships and identity side by side and
exposure and regulatory at full width, and a saved layout that was not
rearranged has them moved the same way.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
GET /api/v2/findings/ ran one dojo_findingvulnerabilityreference query per
finding on the page, because vulnerability_ids is serialized on every row
and FindingViewSet's queryset doesn't prefetch it.

Adding vulnerability_id_prefetch() to get_queryset() would break a subclass
or plugin that adds its own (Django raises "lookup was already seen with a
different queryset"), which is why DefectDojo#16512 left it out. Instead,
FindingViewSet.paginate_queryset() loads the references for the page with
prefetch_related_objects(), skipping findings whose references are already
prefetched, so such a queryset keeps working and nothing is loaded twice.

The list scaling tests no longer leave dojo_findingvulnerabilityreference
out of the count, give each finding two vulnerability ids, check their
order in the response, and cover a queryset that already prefetches them.

Fixes DefectDojo#16652

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0125HXBmURB5ZfBkH3T4LtME
@github-actions github-actions Bot added New Migration Adding a new migration file. Take care when merging. settings_changes Needs changes to settings.py based on changes in settings.dist.py included in this PR labels Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs helm New Migration Adding a new migration file. Take care when merging. parser settings_changes Needs changes to settings.py based on changes in settings.dist.py included in this PR unittests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants