Worker jobs misreport upstream failures #887

Problem

Worker jobs misreport upstream failures.

• Jobs report success when the upstream failed. refresh_clinvar_controls (worker/jobs/external_services/clinvar.py) succeeds with 0 of 12 ClinVar versions processed, and the HGVS job succeeds with every lookup failed.
• A single transient 503 from the ClinGen Allele Registry (lib/clingen/services.py, worker/jobs/external_services/clingen_cache.py) fails the job with a non-retryable category, which stops the rest of the pipeline.
• worker/lib/managers/utils.py classifies every HTTPError as a retryable network error, so deterministic 4xx and 5xx responses retry three times. A dcd-mapping 500 re-runs hours of mapping.
• A transient cdot-rest error during variant validation is reported to the uploader as invalid rows and never retried.

Scope

• A job whose upstream work all failed ends FAILED, and one where some of it failed ends with a partial-success outcome that records the counts.
• Classify HTTP errors by status: 408, 429 and 5xx from idempotent lookups are retryable; other 4xx are not; a dcd-mapping 5xx is not retried automatically.
• Retry transient ClinGen and cdot-rest errors with backoff inside the call before failing the job.
• Report a cdot-rest outage to the uploader as a server error, not as invalid rows.

Acceptance criteria

• Tests assert the ClinVar and HGVS jobs fail when every upstream call fails.
• Tests assert the classification of 404, 429, 500 and 503 responses.
• A test asserts one transient ClinGen 503 doesn't fail the job.
• worker/pipeline_management.md matches the new classification.