[mihow]

Summary

When someone filters the occurrence list and then exports, they expect the file to hold what they were looking at. Until now it did not: the exporter applied only the capture set filter, so date, taxon, algorithm, verification and taxa list filters were silently ignored, and the project's default filters (score threshold, included and excluded taxa) were never applied. The list and the exporter each had their own copy of the filter logic.

This change gives the occurrence list one definition of its filter parameters, and makes the list, the occurrence stats and exports all use it. An export now selects the same occurrences that the occurrence list shows for the same parameters. Export filters are checked when the export is created, so a bad value gets a clear 400 answer instead of a server error or a filter that is quietly dropped. Saved occurrence sets (#1539) will use the same path to store and replay filters.

The occurrence list itself returns the same occurrences as before for every valid query string we measured. The visible changes on the list are in error handling: malformed values that used to cause a server error now get a 400.

List of Changes

Change (effect)	How (implementation)	Notes
1. Exports contain the occurrences the list shows for the same filters: date, taxon, algorithm, job, verification, taxa list, session, station and capture set filters all apply.	The exporter calls filter_occurrences(), which runs OccurrenceFilterSet without a request. The fake-request path (apply_filters) and the exporter's own backend list are removed.	generate_fake_request() stays: the exporters still use it as serializer context to build absolute URLs.
2. New exports apply the project's default filters, as the list does, unless apply_defaults is false. The choice is stored on each export.	DataExportSerializer.validate() stores filters in a normal form that always includes apply_defaults.	Decision for #1537, see "Default filters on exports" below.
3. Exports created before this change run exactly as before when re-run (capture set filter only, no default filters). Stored rows are not rewritten.	DataExport.get_occurrence_filters(): a row without apply_defaults is treated as a legacy row.
4. Export filters are checked when the export is created. A bad value, an unknown parameter or a user-dependent parameter gets a 400 with one message for each field. A non-integer capture set id no longer causes a server error.	clean_saved_filters() validates through the FilterSet's form and compares keys with base_filters.	verified_by_me is refused in saved filters: its result depends on who asks.
5. Equal export filters are stored the same way (ids as numbers, dates as YYYY-MM-DD, booleans, canonical names such as collection_id for collection).	Normal form = the FilterSet form's cleaned_data, made JSON-safe.
6. The export's filter summary names every record it refers to (capture set, taxon, taxa list, algorithm, session, station, device, site, job), with one query per type of record.	filters_display is built by describe_saved_filters(), which reads each filter's model from its field_name and resolves names through ami.base.model_references.	Adds reference_type to Taxon, Deployment, Device, Site and Job. Each entry now has type, id and name; a deleted record has name: null.
7. One definition of the occurrence filters, used by the occurrence list, the occurrence stats endpoint and exports.	The eight custom filter backends became declared filters on OccurrenceFilterSet (method= filters), in a new module ami/main/occurrence_filters.py. Parameter names, the taxon and collection aliases and repeatable algorithm / not_algorithm are kept.	All eight backends moved; none had to stay (see below).
8. Malformed filter values on the occurrence list and the stats endpoint get a 400 instead of a 500.	Typed form fields: IntegerListField, QueryBooleanField (in ami/base/filters.py) and the existing RelatedIdFilter.	Behaviour change, listed below.
9. The API schema documents every occurrence filter (19 parameters; previously 9).	drf-spectacular reads the declared filters and their help text.	No new schema warnings.
10. Removed code that only the old backends used: DateRangeFilterSerializer and FilterParamsSerializer.

Detailed Description

Where the filters live

ami/main/occurrence_filters.py holds:

• OccurrenceFilterSet: the six existing model fields plus the declared filters determination, taxon, collection_id, collection, algorithm, not_algorithm, job, date_start, date_end, verified, verified_by_me, taxa_list_id, not_taxa_list_id. Each filter method builds the same Q expression the old backend built. Declared filters are applied in the same order as the old backends ran.
• OccurrenceFilterForm.clean(): resolves aliases (taxon → determination, collection → collection_id; the canonical name wins when both have a value, as before), checks that date_start is not after date_end, and checks that the taxon exists.
• clean_saved_filters(), filter_occurrences() and describe_saved_filters() for saved filters.

The module is not specific to exports, so occurrence sets can use it. apply_defaults stays outside the FilterSet because apply_default_filters() reads it before the FilterSet runs, as on the list.

Backends that stayed: none. verified_by_me depends on the signed-in user, but a FilterSet receives the request (self.request), so it moved too. It is the one parameter refused in saved filters. A test fails if a parameter is added to the FilterSet without deciding whether saved filters can hold it.

Default filters on exports

Exports now apply the project's default filters unless apply_defaults is false, which matches the occurrence list. Every new export records the choice as apply_defaults in its filters. Exports created before this change have no apply_defaults key; they keep their old behaviour (capture set filter only, no default filters). The export form in the UI does not offer the choice yet, so exports from the UI apply the defaults. Its record count will match what the occurrence list shows with the default filters on, and can be smaller than before for the same capture set.

Measured parity (not inferred)

I ran a matrix of 40 occurrence-list query strings against a fixture of 7 occurrences, on main and on this branch, through the list endpoint and the stats/model-agreement endpoint. The matrix covers every filter parameter, repeated algorithm / not_algorithm, the not_ filters, both aliases, apply_defaults=false, combinations of four filters, and ordering. I captured every SQL statement of each list request (ids replaced by placeholders).

• All 40 valid query strings return the same occurrences and the same stats on both branches.
• The SQL that selects and counts occurrences is identical for all 40.
• The only query that changed: the taxon existence check is SELECT 1 ... LIMIT 1 instead of loading the taxon row.
• 14 malformed query strings: see the behaviour changes below.

The SQL is unchanged, so I did not run EXPLAIN.

Behaviour changes on the occurrence list and stats endpoint

Request	Before	After
Non-integer taxon, determination, collection_id, collection, algorithm, not_algorithm, taxa_list_id, not_taxa_list_id	500	400 with a message for that parameter
taxon or determination with an id that does not exist	404 {"detail": ...}	400 {"taxon": [...]}
verified / verified_by_me other than true, false, 1, 0 (any case)	treated as true	400
algorithm= (blank)	500	no filter
Error text for job=abc and date_start=notadate	DRF wording	Django form wording ("Enter a whole number.", "Enter a valid date.")

A taxa_list_id that does not exist is still ignored, as before.

How to Test

1. Run the tests: docker compose run --rm django python manage.py test ami.main.models_future.tests.test_occurrence_filters ami.exports
   • OccurrenceFiltersTest checks the list against the results measured on main, compares the list with saved filters and with the stats endpoint for each query string, checks every malformed value for a 400, and checks the query counts.
   • ExportFiltersTest checks both export formats against the list, creates exports through the API (normal form, 400s), runs a CSV export with date and taxon filters, and re-runs an export saved before this change.
2. By hand: open the occurrence list for a project, filter by a date range and a taxon, and note the count. Create an export with the same parameters:

   POST /api/v2/exports/
   {"project": <id>, "format": "occurrences_simple_csv", "filters": {"taxon": <id>, "date_start": "2024-06-01", "date_end": "2024-06-30"}}
   

   The export's record_count should match the list's count, and filters should come back with "apply_defaults": true.
3. Send "filters": {"collection_id": "abc"} or {"limit": 10}: the answer should be a 400 with the field named under filters.

Open questions

1. A taxa_list_id that does not exist is ignored (kept from before). For a saved export this means a deleted taxa list makes the export wider. Should it be a 400 instead?
2. The UI export form sends only the capture set and has no "apply default filters" choice. A follow-up could send the current occurrence-page filters and the choice.
3. classification_threshold is documented on the occurrence list, but it has no effect there: apply_default_filters() skips all default filters when apply_defaults=false, and uses the project threshold otherwise. Saved filters refuse it. Should it be removed from the list's documentation?

Closes #1537
Part of #1548

🤖 Generated with Claude Code

https://claude.ai/code/session_01MeUoKCfkiZaZjhja7e6AGh

[netlify]

✅ Deploy Preview for antenna-preview canceled.

Name	Link
🔨 Latest commit	398534b
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6ad979729203f8046fc95ca1

[netlify]

✅ Deploy Preview for antenna-ssec canceled.

Name	Link
🔨 Latest commit	398534b
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-ssec/deploys/6ad97972a8ebd98171a32daa

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mihow]

Claude says: Heads-up from #1561, which adds date, device type and site filters to the captures and sessions lists. It touches the same date filtering code as this PR, so whichever merges second needs a small rebase.

What #1561 changes in the shared code:

• DateRangeFilterSerializer moves from ami/main/api/views.py to ami/base/serializers.py, because captures and sessions now use it. It extends a plain serializers.Serializer and validates itself.
• FilterParamsSerializer is removed there too, as here, because it had no other users.
• OccurrenceDateFilter stays in views.py, but reads validated_data instead of calling .clean().
• The start/end check now allows equal dates, and the message reads "Start date must be on or before end date".

Rebasing this PR after #1561: keep this PR's deletion of OccurrenceDateFilter, and keep DateRangeFilterSerializer in ami/base/serializers.py. The form check here (date_start > date_end) already allows a single day. Its message could say "on or before" to match.

A shared date rule, worth using here. #1561 adds get_date_range_bounds() and a DateRangeFilter backend in ami/base/filters.py. They turn date_start / date_end into a half-open datetime range, from start 00:00 up to but not including (end + 1 day) 00:00, in local capture time. The filter_date_start / filter_date_end methods here cast with detections__timestamp__date. That gives the same result but keeps Postgres from using a timestamp index for the range. On a local copy of production data, a one-week range on the captures table took 17 ms with the datetime comparison and 214 ms with the cast. The occurrence filter goes through the detection join, so its own gain is not measured.

def filter_date_start(self, queryset, name, value):
    return queryset.filter(detections__timestamp__gte=datetime.datetime.combine(value, datetime.time.min))

def filter_date_end(self, queryset, name, value):
    next_day = value + datetime.timedelta(days=1)
    return queryset.filter(detections__timestamp__lt=datetime.datetime.combine(next_day, datetime.time.min))

The matching rule stays the same: an occurrence matches when it has a detection on or after the start and one on or before the end.

Later. Once the FilterSet pattern here lands, captures and sessions could move to their own FilterSets in the same style. #1561 keeps them as filter backends so it does not depend on this PR.