Repository navigation
Record what each retrain did with every occurrence, and the sets it trained and tested on - #1570
Closed
mohamedelabbas1996 wants to merge 26 commits into
Closed
mohamedelabbas1996 wants to merge 26 commits into
mohamedelabbas1996 wants to merge 26 commits into
Conversation
Training is run per project and rewrites what every later identification is compared against, so it belongs behind its own permission rather than any existing run_*_job one. ML data managers get it with the other job permissions they already hold; the migration grants it to the groups that exist today so current managers do not lose the ability when the job appears.
A head trained only on the species someone happened to verify cannot predict the rest, so a region's expected species have to come from somewhere other than the verified data. The project now names that list, and training reads it as the class list. It is optional: without one, the classes are whatever has been verified.
A head is only worth retraining on the crops a person has already named, so this collects verified detections together with the embeddings stored for them and writes one dataset file per run. Decisions worth knowing about: The train/test split is grouped by occurrence, not by detection. Crops from one occurrence are near-identical frames of the same insect, so splitting per detection puts the same animal on both sides and the held-out score flatters the head. It is also a hash of the occurrence id rather than a random draw, so the held-out set stays the same between runs and two heads can be compared. The vector width is read from the stored vectors rather than assumed, because the embedding column is unsized and two algorithms may differ. Species with too few verified crops are dropped, since one example cannot be both trained and evaluated on. The classes come from the project's taxa list where it has one, so the head covers the region and not only what has been verified, and the classes with no verified data are recorded in the file rather than silently left out. The metadata is declared as a schema instead of assembled as a dict, so the shape is checked once here rather than at every reader, and the processing service echoes it back for the job to store. The rows are also served over the API, paged at a limit set for their size: a 1024-dimension vector is roughly 20 KB of JSON, so the platform default of 10 is useless and no limit at all returns hundreds of megabytes.
The training job form asks for a split ratio and an optional occurrence set. Both questions are unanswerable without knowing what is there, so the summary endpoint now answers for exactly the run being set up. It takes the occurrence set, or reports every verified occurrence when none is chosen, which is what a run without a set does. It defaults the settings to the algorithm's own training config rather than to module defaults, so the first numbers someone sees are the numbers a run would use, and it accepts the settings as parameters so the form can preview a changed ratio without starting a job. The same bounds a job is held to are applied here, since stats computed under settings no run could use are worse than no stats. It also reports the classes that survive min_per_species, and names the species dropped: a raw class count overstates what the head would come out knowing. While adding the set parameter: the endpoint was gated on project visibility, which a non-draft project grants to everyone, so any account could read every verified label in a project together with the vector for each crop. It is now gated on the same permission as the training job.
feat/retrain-job-plumbing was cleaned up and rebased onto the current feat/detection-embeddings-task, which moved DEFAULT_EMBEDDING_KEY out of ami.main.models and added its own 0098/0099 migrations in ami.main. Update the training dataset builder's import and renumber this branch's migrations (0098-0102 -> 0100-0104) to follow the new leaf. Co-Authored-By: Claude <noreply@anthropic.com>
embedding.vector comes back as a pgvector.HalfVector in some query paths and a plain list in others, which is exactly what as_half_precision() in ami/ml/models/embedding.py already normalizes; writer.py, reader.py and same_vector() all go through it. dataset.py called .to_list() directly instead, which breaks depending on which shape the query path hands back; hit live in a local demo run where build_training_dataset() failed at its very first stage. Co-Authored-By: Claude <noreply@anthropic.com>
feat/detection-embeddings-task dropped the per-vector key, deleted ami/ml/embeddings/reader.py, and replaced stored_length() with a declared Algorithm.embedding_dimensions field. Updates: - verified_training_rows(): drop the key= filter, DEFAULT_EMBEDDING_KEY no longer exists - dimensions read from algorithm.embedding_dimensions instead of DetectionEmbedding.objects.stored_length() - as_half_precision() was deleted along with it (Michael's own code never reads vectors back, so he had no further use for it); added a small local equivalent, _vector_as_array(), since we still do - test fixture needs embedding_dimensions set now that it is read from the algorithm instead of inferred from stored vectors; renamed the one test whose docstring described the old (now wrong) behavior Co-Authored-By: Claude <noreply@anthropic.com>
"Not Identifiable" (rank UNKNOWN) and "Not Lepidoptera" (an ORDER) were trained as classes of a species head. Verified occurrences now count only when their determination is at species rank, which also keeps the missing-embedding count consistent with what a run can use. Co-Authored-By: Claude <noreply@anthropic.com>
Adds the training job: it builds the dataset, hands a processing service the URL, and records what comes back as a new algorithm version. Training outlasts the request that starts it, so the service reports its result to a callback instead of the connection being held open. The service has no Antenna account, so the callbacks are authorised by a signed token issued when the job was dispatched, and the job is looked up directly because project visibility would refuse an unauthenticated caller. The weights are uploaded the same way: without that, the head exists only in a cache directory on the service's disk and Antenna records a version it cannot point at. An upload is refused once the job has finished, since the token lasts a day and the path is fixed by the job, so otherwise the weights behind a registered version could be swapped out after the run ended. A job type now also declares what it needs before it can start and whether a user may create it at all, rather than each job discovering a missing param partway through its own run.
The refactor that gathered the training modules into one package left two imports behind, so running a training job raised ImportError on ami.ml.training_dataset instead of building a dataset. The tests did not catch it because neither import is reached until a job actually runs.
Requiring a set meant that retraining on everything verified so far - the ordinary case - first made someone save a set of everything, which is a step with no decision in it. A job without a set now learns from every verified occurrence in the project. The dataset file records which set was used, or that there was none, so a run is still legible afterwards either way. Naming a set is still worth doing when it matters: a set is fixed once created, so that run can be repeated.
A key is shared by every version of an algorithm, while embeddings are stored against one version row. Taking whichever row the database returned first could pick an older version and train on an empty set of vectors, or on vectors from a different fitting, with nothing in the job to say which had happened.
Training is the long stage of the job and the service says nothing while it runs, so the bar sat at nought for as long as the fitting took and a person could not tell a slow run from a dead one. The service is now told where to report, and posts the epoch it has reached. The callback is authorised by the same signed token as the result and the head upload, since a service has no Antenna account. The epoch and the total land as stage parameters, and the stage progress is the ratio of the two. The total is written at dispatch from the run's own settings, so the stage reads "0 of 5000" while the service is still starting rather than an empty pair. A ping never finishes the stage: the result does, because the service still has to score the head and upload it after the last epoch. Pings that arrive late or out of order are dropped rather than dragging a run backwards. Reporting is optional. A service that does not post anything behaves exactly as before, and a ping that fails to send must not fail the run.
The training request was a dict literal and the result was read with .get() at a dozen call sites, so neither side of the exchange was written down anywhere. Every other service contract in this module is a schema; this one was the exception, and a service renaming a field would have produced an empty metrics dict and no error. TrainingRequest and TrainingResult mirror the service's own TrainRequest and TrainResponse, the way PipelineRequest and PipelineResultsResponse already do for processing. TrainingResult keeps unknown fields: a newer service may report more than this one knows about, and the result is the only record of what a run did. incumbent_metrics is nullable because a service reports null when there was no current head to score, which is exactly what a first retrain does.
… to read A result could arrive two ways: in the body of the /train response, or at the job's callback. The two carried the same outcome in different shapes, and both of these bugs lived in that gap. On the callback path the dataset metadata the service echoes was read as payload["dataset"]["metadata"], which is not where it is, so it was never parsed and a registered version never recorded which occurrence set it learned from. The provenance this is all for was quietly dropped on every real run. On the inline path the same code would have raised AttributeError, because there "dataset" is the schema object rather than a dict. It never fired only because a service posts its callback first and the duplicate guard returned early. /train is now an acknowledgement: the result arrives at the callback, is validated by a serializer at the view, and record_result takes the parsed result rather than a raw payload. One shape, one path, parsed once at the boundary. The result is validated because the new algorithm version is built from it. The echoed dataset metadata is read leniently: a service echoing an older shape should still have its result recorded, and the cost is a version that cannot say which set it came from, not a head that is lost. Also fixes a flaky test: with a handful of rows, a hash split does not give a particular ratio a particular count, so the ratio is now compared across two values rather than asserted exactly.
Every training job read as "internal", the mode for work the platform does itself, because Job.setup() decides the mode from the pipeline and a training job has none. It is the platform's own word for what this job has always done: hand the work to a service and wait to be called back.
TrainClassifierJobConfig names the head (algorithm_key), the species list (taxa_list_id) and the data the head learns from: this project's verified occurrences, optionally one set, and offline iNaturalist data, which is declared but refused until the processing service can train from it. A retrain with no data source, or a set with project occurrences off, is a 400 when the job is created. Fitting overrides such as test_fraction pass through and are range-checked by AlgorithmTrainingConfig, which keeps owning their defaults. The job reads its settings through the schema instead of raw params lookups. Co-Authored-By: Claude <noreply@anthropic.com>
…er its pipeline
When the service reports the pipeline that serves the new head, the
version is registered under that key at version 1, which is how the
service reports it in /info, and the service's pipelines are registered
again for the project. The retrained head then appears as a pipeline that
can process captures, instead of an algorithm no pipeline uses. The
lineage ("retrained v2") moves into the name. Services that do not report
a pipeline keep the previous key scheme.
Co-Authored-By: Claude <noreply@anthropic.com>
Registering the service's pipelines inside the training callback asked the service for /info while it was still waiting on that same callback, and the request timed out, so the new head was recorded but never offered. The registration now runs in a task queued on commit, retried while the service is busy. Co-Authored-By: Claude <noreply@anthropic.com>
…roject data - use_offline_inat is no longer refused. With project occurrences on, the service adds its offline vectors to Antenna's training set; with them off, no data leaves Antenna and the service gets the species list (classes) instead of a dataset URL. TrainingRequest gains use_offline_inat and classes, and dataset_url becomes optional. - Fix: a retrain that named a taxa list crashed, because TaxaList has no for_project(); the list is now scoped through its projects. - A retrained version copies its parent's embedding_dimensions: same frozen backbone, and without it the vectors its pipeline sends are refused. Co-Authored-By: Claude <noreply@anthropic.com>
The job form only made processing jobs. Retraining had to be posted by hand, which put it out of reach of the people the feature is for. The form now asks for the job type first, and the processing fields and the retraining fields are the two branches of that choice. Retraining asks for the algorithm, an optional occurrence set, and the two settings worth changing per run. The occurrence set is optional and empty means every verified occurrence in the project, which is the ordinary case. A split ratio is unanswerable without knowing what is behind it, so the form reports what the run would learn from - crops, occurrences, species, and how the split falls - and re-reads it as the set or the ratio changes. Verified crops with no embedding from the chosen algorithm are called out, since they are silently left out of the run.
Leaving a setting empty means "use the algorithm's own", but the empty field was sent as an empty string, which the job then tried to read as a number and failed on at run time. Empty fields are left out of the params instead. A ratio outside the valid range is also no longer asked about: the field already reports it, and the summary request behind it could only come back as an error.
The list reads its type labels from a map that the new type was missing from, so every retraining run showed a blank Type while the API was already naming it.
The retrain form gains "Training data" (this project's verified occurrences, the same plus offline iNaturalist images, or offline iNaturalist images alone) and a species list picker. Offline data alone hides the project summary, since no project rows are used, and needs a species list. Co-Authored-By: Claude <noreply@anthropic.com>
The result registry only collected kinds from post-processing tasks. A job type now declares result_models the same way, and the registry reads both, still refusing one kind declared by two different data models. Co-Authored-By: Claude <noreply@anthropic.com>
When a retrain builds its training set, it now writes down every verified occurrence it considered: - a classifier_training algorithm result per occurrence: its split, whether it was trained on, or why it was left out (not in the taxa list, too few crops for its species, no embedding), and the label it was given; - the train and test splits as two occurrence sets named after the job, kept in the job's result and on the registered version's training_info, so a run can be repeated and its test set reused for evaluation. The service may report per-species train accuracy (optional species_train_accuracy on the training result); it becomes each result's value. No service sends it yet, so the value stays empty for now. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
✅ Deploy Preview for antenna-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
mohamedelabbas1996
force-pushed
the
feat/retrain-job-form
branch
from
October 30, 2026 05:20
6674160 to
cd3f048
Compare
This was referenced Oct 30, 2026
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A retrain now leaves a record, per occurrence, of what it did with it: whether the occurrence was trained on or held out for testing, whether it was left out and why, and the species it carried at the time. It also saves the exact occurrences it trained and tested on as two occurrence sets, so a run can be repeated and its test set reused to score other models. Stacked on #1504.
List of Changes
not_in_taxa_list,below_min_per_species,no_embedding), and its determination at training timeclassifier_training(ClassifierTrainingResultData, aDeterminationSnapshot), written in bulk when the training set is built (ami/ml/training/results.py)JobType.result_models;result_data_models()reads both registries and keeps the "two models for one kind" checksave_split_sets()(the only place that writes sets, to keep the move to #1543 small);AlgorithmTrainingInfo.train_/test_occurrence_set_id/_namespecies_train_accuracyonTrainingResult, copied onto each row'svalueat the callback; the service does not report it yet, so values stay emptyDetailed Description
Rows are written when the training set is built, not when the result returns: the label at training time and the reject reasons are only known then, and rebuilding them later would mislabel anything re-identified while training ran. The cost is that a run that never reports back leaves rows with no accuracy. Rows point at the algorithm that was retrained; the new version links to them through
training_info.job_idand the set ids. No UI yet: the occurrence history skips kinds it does not know.How to Test
python manage.py test ami.jobs ami.ml(402 tests OK locally,makemigrations --checkclean).ami/ml/test_training_results.pycovers the new kind and the sets.