[mohamedelabbas1996]

Summary

Retraining a classifier for the whole platform needs every verified species identification in Antenna, not one project at a time. This PR adds a periodic export that writes the verified species detections of every project that opted in into versioned Parquet files: a manifest (labels, split, box, crop and capture locations) and one vector file per algorithm. Projects share nothing unless a superuser turns on their share_verified_data_for_training feature flag. The weekly schedule is created switched off.

The selection rule is the one the per-project training set already uses (verified_occurrence_ids in ami/ml/training/dataset.py): a non-withdrawn identification, on an occurrence determined to species. The export does not use occurrence sets, so it is not affected when the retraining stack moves to #1543.

Stacked on #1504 (feat/retrain-job-form, at 70934961). Design notes: ~/antenna/reviews/training-data-export-design.md (to be posted on the retraining epic).

List of Changes

User / operator effect	Implementation
A project can opt in to share its verified data for platform training. Off by default.	ProjectFeatureFlags.share_verified_data_for_training: bool = False (ami/main/models.py)
Superusers can start an export from the admin and see what each snapshot holds, per project.	TrainingDataSnapshot and TrainingDataSnapshotProject (ami/ml/models/training_data.py), admin with a per-project inline; saving a new snapshot queues the export
One file lists every verified species detection of the opted-in projects, with its label and train/test split.	ami/ml/training/export.py: manifest.parquet. Rows streamed with a server-side cursor and written in row groups, so memory stays flat. Split by occurrence with the existing split_for
Vectors come in one file per algorithm, never mixed.	vectors/algorithm=<key>/part-00000.parquet: detection_id + fixed_size_list<halffloat>[dims]. A vector with the wrong length is skipped and counted
Each snapshot says how it was made.	metadata.json: rules, files, algorithms, per-project counts. The same counts are stored on the snapshot rows
Exports can run weekly. Off until an admin enables it.	Celery task ami.ml.tasks.export_training_data_snapshot; data migration adds a django_celery_beat PeriodicTask (Sunday 03:00, enabled=False)
Two exports never run at once.	Postgres advisory lock; a second run raises SnapshotAlreadyRunning
Storage stays bounded.	Files of successful snapshots beyond the newest TRAINING_DATA_SNAPSHOTS_KEPT (default 4, env setting) are deleted; their counts stay
New dependency	pyarrow==17.0.0 in requirements/base.txt

Detailed Description

Files in a snapshot

training-data/snapshot-<id>/
  manifest.parquet
  vectors/algorithm=<key>/part-00000.parquet
  metadata.json

Manifest columns: detection_id, occurrence_id, source_image_id, deployment_id, project_id, site_id, device_id, taxon_id, taxon_name, taxon_rank, identification_id, identified_at, split, bbox, captured_at, crop_url, capture_path, capture_url.

• identification_id / identified_at are the latest non-withdrawn identification.
• capture_url is built from the capture's public base URL only. Presigned URLs are not written, because they expire long before a snapshot is read; capture_path is always there.
• No user names or emails are exported.

Why Parquet and not image shards

Head retraining reads stored embeddings, so the vectors and labels are what it needs. Parquet is built from SQL alone, is much smaller than crops, and any common tool reads it. Image shards (WebDataset) can be built later from the manifest, which keeps crop and capture locations. Size estimates are in the design notes; they are estimates, not measured.

Queries

The number of queries depends on the number of algorithms, not on the number of detections. A test pins this with the query cache off, at 1 and 7 rows.

Not in this PR

• A read API and model-level permissions for a "ML curators" group (needs Extend permissions framework to support model-level permissions #1010). Only superusers see snapshots, through the admin.
• A project-manager-facing setting for the opt-in. Today it is a feature flag that superusers set.
• A per-project Parquet format in the data-export menu.
• Image shards (WebDataset).
• Linking a snapshot to frozen occurrence sets, once occurrence sets can be frozen (Save and reuse sets from list views (captures, occurrences, taxa) #1548).

Tests

Measured on 2026-10-30, in the CI compose stack: makemigrations --check --dry-run reports no changes, and manage.py test ami.exports ami.ml ami.jobs ran 417 tests, OK (2 skipped). The 8 new tests are in ami/ml/test_training_export.py:

• an opted-out project's rows and vectors are left out;
• only species-rank determinations with a non-withdrawn identification are exported (genus, "Not Identifiable", unidentified and withdrawn are left out);
• the exported occurrences equal verified_occurrence_ids for the same project;
• each vector file holds one algorithm's detections and values only;
• the query count is equal at 1 and 7 rows, with the query cache off;
• the manifest, a vector file and metadata.json read back with pyarrow and json;
• no opted-in project gives an empty, successful snapshot;
• pruning removes old files and keeps the counts.

How to Test the Changes

docker compose -f docker-compose.ci.yml run --rm django python manage.py test ami.ml.test_training_export
docker compose -f docker-compose.ci.yml run --rm django python manage.py makemigrations --check --dry-run

By hand: in the Django admin, turn on share_verified_data_for_training in a project's feature flags, then add a Training data snapshot. When it finishes, its files are listed on the snapshot and under training-data/ in media storage.

Base (2026-10-30)

Retargeted from #1504 to #1502: the export only needs #1502's rule for which occurrences count as verified species, so it can merge without waiting for the training job (#1503) or the form (#1504).

[netlify]

✅ Deploy Preview for antenna-preview ready!

Name	Link
🔨 Latest commit	8693bf9
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6ae42a0ae3627b8a40b88255
😎 Deploy Preview	https://deploy-preview-1571--antenna-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	1 paths audited Performance: 55 (🔴 down 10 from production) Accessibility: 82 (🔴 down 7 from production) Best Practices: 92 (🔴 down 8 from production) SEO: 92 (no change from production) PWA: 80 (no change from production) View the detailed breakdown and full score reports
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 946c8e33-5b59-4018-905c-e99e8419260f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}