Repository navigation
Fix/finding api vulnerability id prefetch - #16658
Closed
jensholdgaard wants to merge 26 commits into
Closed
jensholdgaard wants to merge 26 commits into
jensholdgaard wants to merge 26 commits into
Conversation
…ter-into-dev-3.4.304 Release: Merge back 3.4.304 into dev from: release/merge-master-into-dev-3.4.304
…tDojo#16547) * feat(locations): LocationData builders for domain, ip and cert Parsers can report a DNS name, an IP address or an X.509 certificate as a typed location: LocationData.domain(), LocationData.ip() and LocationData.cert(). The builders emit a fixed set of keys so every consumer reads one shape; normalizing the values is the consumer's job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(locations): read a single SAN as one name and document the cert and ip rules Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#16637) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…jo#16632) * perf(qualys): look up QID details once per report, not once per finding The Qualys XML parser found each finding's QID details with a search from the document root, which walks every host before it reaches the glossary. Parsing cost findings x report size: a 56-host, 2,408-finding report took 40.0 s of CPU to parse, against 2.0 s for 7 hosts. The glossary is now indexed once per report and each finding reads its entry from that index. The first entry for an id still wins, as the search returned it, and the parsed findings are identical to before on reports of 7 to 56 hosts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(qualys): count per-QID lookups wherever the search starts The guard only watched searches made on the whole tree, so a lookup made from the root element once per finding would have brought the quadratic parse back without failing it. Every find, findall, findtext and iterfind, on a tree or an element, resolves its path through ElementPath.iterfind, so that is what the test counts now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…nd records screenshots (DefectDojo#16639) - API v2 Sample Code: query /api/v2/findings/ (readable with any token) instead of /api/v2/users/, print the results instead of the Response internals, show the paginated response shape and a filtered query, with placeholder host and token. - do_not_reactivate: a reimport keeps a closed finding closed; new and open findings are processed as usual. - Location page: its table lists open findings only; say how to reach every finding on a Location through the Total Findings count. - Location Map: link it from the Locations overview, and state what a Reader can and cannot change. - Recapture the upstream records screenshots with the row menu. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…le page (DefectDojo#16614) * docs(navigation): describe the Preferences page and its landing page Add a Preferences page for DefectDojo Pro: the default landing page applied after sign-in, the default dashboard, saving, and the feature flag. The sidebar page names Preferences in the account menu, and the custom dashboards page points at it for changing the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(navigation): call the fallback page Home and cover identity-provider sign-ins Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(navigation): the default landing page is set on the User Profile page The landing page moves from a Preferences page to a Signing In field on User Profile, and the default dashboard stays on the Dashboards page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e tiles (DefectDojo#16640) The Asset page's Finding Distribution panel is now a ring of Active, Mitigated and Other findings, whose legend links to each status, beside the static and dynamic shares. English page only; the language copies are translated automatically. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…es how common a tag is (DefectDojo#16634) * perf(findings): resolve tag filter names to tag ids before matching tag_exists() matched the tag through a join on the tag table's name, so the planner could not tell which tag a query was about and assumed the average number of rows per tag. For a tag carried by most findings that estimate was off by three orders of magnitude (573 expected, 400,001 actual), and the finding list's COUNT and id queries took a nested-loop semi-join that probed every tagged row: 20.6 s for one COUNT over 400k tagged findings, 2.5 s once the subquery names the tag id. The matching tags are now read first (one indexed lookup on the tag table) and the EXISTS subquery tests the through table against those ids. A lookup that matches more than TAG_ID_LITERAL_LIMIT tags (a short icontains value) keeps the join. Results, counts and paging are unchanged: the existing equivalence tests against the old JOIN form pass as they are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(findings): resolve every AND-ed tag in one read, and match nothing for an unknown one tags__and (and its test__, test__engagement__ and test__engagement__product__ forms) looked each listed tag up on its own, one query per tag. The names are now resolved in one name__in read, and one EXISTS per tag id is built from it. A listed name that is no tag cannot be present, so the filter returns an empty queryset without building the subqueries. Tests: the over-limit fallback is checked on the rows it returns, not only its SQL, and an unknown tag is checked through the FilterSet for tags__and, test__engagement__tags__and and not_tags. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…DefectDojo#16642) Describe the enterprise option to run Sensei on a Cloud instance's own model provider account through AI Model Settings, how it differs from on-premise (keys required, public base URL only, no fallback to DefectDojo's model), and point readers to hello@defectdojo.com to request it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(models): add a default-queryset hook for plugins Adds dojo.query_hooks: a registry of default-queryset filters keyed by model, applied by the HookedManager declared as `objects` on Product_Type, Product, Engagement, Test, Finding and the rows that hang off them. With nothing registered the querysets are unchanged. Each of those models also gets an explicit `unfiltered` manager, and two nullable columns (archived_at, archive_batch) that open source never reads or writes, with partial indexes built concurrently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(models): renumber the queryset-hook migrations after dev Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: expect the archive columns on endpoint status rows test_endpoint_queries compares Endpoint_Status.values() against literal rows, which now include the two nullable columns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(migrations): hold the archive-column locks only for the SQL itself Measured on 10 million findings, 0304 blocked every read of dojo_finding for about 40 seconds. Its SQL takes milliseconds, but Django rebuilds its model state after each of the migration's operations, and each table's ACCESS EXCLUSIVE lock is taken by its ALTER and held through all the rebuilding that follows until the migration commits. The operations are now applied to the migration state only, and the database work is the SQL they generate (from sqlmigrate), frozen and run as one batch at the end. The resulting schema is identical; the slowest concurrent read during the migration dropped to 0.4 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(dedupe): let a plugin set excess duplicates aside instead of deleting them register_excess_duplicate_handler installs one handler for the chunks the excess-duplicate task removes. It runs inside the chunk's transaction with the rows locked; when it returns True nothing is deleted. With no handler, or one that returns False, the chunk is deleted as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…jo#16563) * docs(asset-modelling): document Archive and retention rules Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(asset-modelling): say archived data still counts toward the license Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(archive): archiving and restoring run in the background The object disappears at once and the rest of its tree is archived in the background; restores work the same way. A bulk selection or one Triage Engine run is one archive, restored as a whole. Archive and unarchive return 202. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(archive): say what stays visible while archiving, and the parent rule on restore Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(archive): archived data does not count against the license; excess duplicates are archived Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(archive): archiving works under a breached license, restoring does not Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…api/v2/findings/ (DefectDojo#16644) * fix(findings-api): honor ?engagement=, ?product= and ?prod_type= on /api/v2/findings/ A Finding has no engagement, product or prod_type field and ApiFindingFilter declared no filter by those names, so django-filter dropped the parameters and the list answered with every finding the caller could see. Declare them as short names for the test__engagement, test__engagement__product and test__engagement__product__prod_type filters, and document them on the Findings pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(findings-api): pin that the short names do not widen for a product member; note the change in the 3.5 upgrade page Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…16647) The built-in API Importer role holds no Note permissions, so the chart should not show it adding, editing or deleting notes. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…plates and the mandatory-field error (DefectDojo#16574) * docs(connectors): document ServiceDesk Plus custom fields, ticket templates and the mandatory-field error * docs(connectors): ServiceDesk Plus Ticket Template sets the subject, and fallbacks are listed under Errors
…fectDojo#16626) Maps a comma-separated DD_SECRET_KEY_FALLBACKS to Django's SECRET_KEY_FALLBACKS, so an operator can change DD_SECRET_KEY and keep values encrypted under the old key readable. The restoring page now says the SSO, OIDC, LDAP and email secrets use DD_SECRET_KEY too, and how to change the key.
…ojo#16645) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* docs(connectors): Security Hub ECR images in use only Documents the ECR Images in Use Only toggle of the Security Hub connector, its Amazon Inspector permission and its limits. Names the form section Import Configuration, as the form shows it. * docs(connectors): Security Hub ECR in-use Sync cost and scan on push * docs(connectors): Security Hub ECR in-use limits, save check and skip scope * docs(connectors): Security Hub ECR in-use source link, scanning per region and multi-arch images * docs(connectors): Security Hub ECR in-use reads Inspector at least once per Record * docs(connectors): Security Hub ECR in-use findings can stay open up to two days after an image stops * docs(connectors): Security Hub records Inspector cannot see now fail instead of skipping Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…fectDojo#16651) * docs(onprem): document the web-request statement_timeout and its settings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(onprem): statement timeout on Kubernetes, poolers and the capped long bound Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(onprem): long statement timeout default when harakiri is unset Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…Dojo#16578) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…jo#16650) Login Settings gains Idle Timeout, and Session Age is now an absolute lifetime counted from the login. Adds a Session expiry section and corrects who can change the page (the Tuner Edit permission). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ojo#16654) The page layouts and EU CRA pages said the Organization Memberships, Asset Identity, Exposure and Regulatory cards sit at the top of the Asset page and that the shipped layout places them first. The shipped layout now starts with the Open Finding Severity meter and places the cards after the Details row, memberships and identity side by side and exposure and regulatory at full width, and a saved layout that was not rearranged has them moved the same way. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
GET /api/v2/findings/ ran one dojo_findingvulnerabilityreference query per finding on the page, because vulnerability_ids is serialized on every row and FindingViewSet's queryset doesn't prefetch it. Adding vulnerability_id_prefetch() to get_queryset() would break a subclass or plugin that adds its own (Django raises "lookup was already seen with a different queryset"), which is why DefectDojo#16512 left it out. Instead, FindingViewSet.paginate_queryset() loads the references for the page with prefetch_related_objects(), skipping findings whose references are already prefetched, so such a queryset keeps working and nothing is loaded twice. The list scaling tests no longer leave dojo_findingvulnerabilityreference out of the count, give each finding two vulnerability ids, check their order in the response, and cover a queryset that already prefetches them. Fixes DefectDojo#16652 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0125HXBmURB5ZfBkH3T4LtME
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We don't want to waste your time, so if you're unsure whether your hypothetical enhancement meets the criteria for approval, please file an issue to get pre-approval before beginning work on a PR.
Learn more here: https://github.com/DefectDojo/django-DefectDojo/blob/master/readme-docs/CONTRIBUTING.md#submission-pre-approval
Description
Describe the feature / bug fix implemented by this PR.
If this is a new parser, the parser guide may be worth (re)reading.
Test results
Ideally you extend the test suite in
tests/anddojo/unitteststo cover the changed in this PR.Alternatively, describe what you have and haven't tested.
Documentation
Please update any documentation when needed in the documentation folder)
Checklist
This checklist is for your information.
dev.devbranch.Extra information
Please clear everything below when submitting your pull request, it's here purely for your information.
Moderators: Labels currently accepted for PRs:
Contributors: Git Tips
Rebase on dev branch
If the dev branch has changed since you started working on it, please rebase your work after the current dev.
On your working branch
mybranch:In case of conflict:
When everything's fine on your local branch, force push to your
myOriginremote:To cancel everything:
Squashing commits
pickbyfixupon the commits you want squashed outpickbyrewordon the first commit if you want to change the commit messageForce push to your
myOriginremote: