[mohamedelabbas1996]

The job that retrains a head: it builds the training set, hands a processing service the URL, and records what comes back as a new algorithm version.

Split out of #1494. The service side is ami-data-companion#167.

What a run does

1. builds the dataset (the PR below this one) and writes it to storage
2. posts the URL to the service and leaves the job STARTED
3. the service trains, uploads the head, reports its epochs, then reports the result
4. the result is registered as a new algorithm version recording what it learned from

The occurrence set is optional. Without one a run learns from every verified occurrence in the project, which is the ordinary case; naming a set makes the run repeatable, and either way the dataset file records which it was.

The interface

A retraining run reports back the way an ML job's results already do: a POST to a route on the job, typed with a schema and a serializer, acknowledged with a typed response. TrainingRequest and TrainingResult mirror the service's own TrainRequest and TrainResponse, as PipelineRequest and PipelineResultsResponse already do for processing. The job is marked ASYNC_API, which is what it has always been.

One way in. /train returns 202; the result arrives only at the callback and is parsed once. Antenna previously also read a result from the /train response body, which gave one outcome two shapes - and both of the bugs this fixes lived in that gap:

• on the callback path the echoed dataset metadata was looked for under a key it is not at, so it was never parsed and a registered version never recorded which occurrence set it came from
• on the inline path the same code would have raised AttributeError; it never fired only because the duplicate guard returned first

The result is validated because the new version is built from it. The echoed metadata is read leniently: a service echoing an older shape should still have its result recorded, and the cost is provenance, not a lost head.

Three things here have no precedent in the platform:

1. The callback token. Antenna hands the service a URL and a token signed for that one job, valid 24 hours. The existing async path does it the other way round: the ADC worker holds an Antenna API token and posts as itself. I kept the signed token because it is the tighter of the two, and because POST /jobs/{id}/result/ resolves to a result_ml_job permission that does not exist, so that endpoint effectively needs a superuser. Happy to switch if you would rather have one mechanism. That permission looks unintended and may deserve its own fix.
2. The head upload. No endpoint existed for a service to send a file back.
3. The progress ping. Processing infers progress from results arriving; one long fit has nothing to infer from, so the service reports its epoch and the stage follows it.

Progress while it trains

The total is written at dispatch from the run's own settings, so the stage reads "0 of 5000" while the service is still starting. A ping never finishes the stage - the result does, since the head still has to be scored and uploaded. Late or out-of-order pings are dropped. Reporting is optional: a service that posts nothing behaves as before.

A run, end to end

Against the BioCLIP 2.5 service on the GPU VM, over a project's real verified crops:

Training set: 96 verified crops over 17 species (74 train / 22 test)
Species list comes from taxa list 'NF retraining list (demo)'
7 species in the list have no verified crops yet
Registered #67 "BioCLIP 2.5 + LogReg head (Newfoundland, 749 species)" v22 as version 22
New head top-1 0.636 does not beat current 0.875 by more than 0.000.

The last line is the point of keeping the score: a retrain that is worse than what it would replace says so rather than quietly taking over.

Stack

#1462 embeddings  ->  #1501 plumbing  ->  #1502 training set  ->  this  ->  job form

Update (2026-10-30)

Tested end to end on a dev instance against a copy of production data (Atlantic Forestry Centre, BioCLIP 2.5 Newfoundland head): retrain from the job -> service fits a head -> new version recorded with its scores -> its pipeline appears in the project -> a processing job classifies captures with the retrained head.

Change	Why
TrainClassifierJobConfig (ami/ml/training/config.py) is the job type's config_schema	algorithm_key, taxa_list_id and the data sources (use_project_occurrences, occurrence_set_id, use_offline_inat) are checked when the job is created. Offline iNat data is declared but refused until the service can train from it. Fitting overrides pass through and are range-checked by AlgorithmTrainingConfig, which keeps owning their defaults.
The new version is registered under the key the service serves the head under (pipeline_slug in the result), at version 1	Pipelines find algorithms by key and version, so registering the service's pipeline reuses this row instead of creating an unrelated algorithm. The lineage ("retrained v2") moves into the name.
The service's pipelines are registered again after the callback, in register_retrained_head_pipeline	So the retrained head can be run straight away. It runs after the callback has answered: calling /info inside the callback deadlocked, because the service is waiting on that answer.
Tests	API refusals for the config; key and version of a served head; the pipeline task is queued on commit and registers for the job's project.

Needs RolnickLab/ami-data-companion#167 at 2f6ce7c or later (reports pipeline_slug, lists new heads in /info).

What each retrain records (folded in from #1570, 2026-10-30)

Summary

A retrain now leaves a record, per occurrence, of what it did with it: whether the occurrence was trained on or held out for testing, whether it was left out and why, and the species it carried at the time. It also saves the exact occurrences it trained and tested on as two occurrence sets, so a run can be repeated and its test set reused to score other models. Stacked on #1504.

List of Changes

User effect	Implementation
Each occurrence a retrain looked at gets an algorithm result: split (train / test), used for training, rejected + reason (not_in_taxa_list, below_min_per_species, no_embedding), and its determination at training time	New result kind classifier_training (ClassifierTrainingResultData, a DeterminationSnapshot), written in bulk when the training set is built (ami/ml/training/results.py)
Job types can write algorithm results, not only post-processing tasks	JobType.result_models; result_data_models() reads both registries and keeps the "two models for one kind" check
"Job #N train set" and "Job #N test set" occurrence sets are saved with every run, and named on the new algorithm version	save_split_sets() (the only place that writes sets, to keep the move to #1543 small); AlgorithmTrainingInfo.train_/test_occurrence_set_id/_name
Per-species training accuracy can be shown when the service reports it	Optional species_train_accuracy on TrainingResult, copied onto each row's value at the callback; the service does not report it yet, so values stay empty

Detailed Description

Rows are written when the training set is built, not when the result returns: the label at training time and the reject reasons are only known then, and rebuilding them later would mislabel anything re-identified while training ran. The cost is that a run that never reports back leaves rows with no accuracy. Rows point at the algorithm that was retrained; the new version links to them through training_info.job_id and the set ids. No UI yet: the occurrence history skips kinds it does not know.

How to Test

python manage.py test ami.jobs ami.ml (402 tests OK locally, makemigrations --check clean). ami/ml/test_training_results.py covers the new kind and the sets.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 0c41b302-5eec-46f4-a7ca-e8065a0419f0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[netlify]

✅ Deploy Preview for antenna-preview canceled.

Name	Link
🔨 Latest commit	4a03bb0
🔍 Latest deploy log	https://app.netlify.com/projects/antenna-preview/deploys/6ae42929ba6a662940819c8c