Skip to content

fix(dry_run): skip MLflow app creation and fix evaluator base_job_name - #6100

Open
amazeAmazing wants to merge 1 commit into
aws:master-nova-follow-upsfrom
amazeAmazing:fix/dry-run-mlflow-instance-type
Open

fix(dry_run): skip MLflow app creation and fix evaluator base_job_name#6100
amazeAmazing wants to merge 1 commit into
aws:master-nova-follow-upsfrom
amazeAmazing:fix/dry-run-mlflow-instance-type

Conversation

@amazeAmazing

@amazeAmazing amazeAmazing commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Issue #, if available: N/A

Description of changes:

fix(dry_run): skip MLflow app creation and fix evaluator base_job_name

Problem

  1. dry_run=True took 5-10 minutes when no MLflow app existed because
    _create_mlflow_config() triggered app creation and polled every 60s
    with a 600s timeout — before the dry_run check could short-circuit.

  2. BenchMarkEvaluator and CustomScorerEvaluator._evaluate_hyperpod() hardcoded job name as
    the job name instead of respecting self.base_eval_name.

Changes

  • _resolve_mlflow_resource_arn and _create_mlflow_config accept dry_run.
    When dry_run=True, they list existing apps but skip creation and waiting.
  • All trainers (SFT, DPO, RLVR, RLAIF, MultiTurnRL) forward dry_run to
    MLflow resolution.
  • MultiTurnRLTrainer.train() now supports dry_run=True.
  • _evaluate_hyperpod() is consistent with SMTJ path.
  • Removed duplicate import time in finetune_utils.py.
  • Removed stray print() in dpo_trainer.py.

Testing

  • 9 unit tests for MLflow dry_run (skip creation/wait/upgrade, warnings, flag forwarding)
  • 2 unit tests for MultiTurnRLTrainer dry_run (skips Job.create, forwards flag)
  • Integration: dry_run completes in ~7s in ap-northeast-1 (zero MLflow apps,
    previously 5-10 minutes)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@amazeAmazing
amazeAmazing marked this pull request as ready for review July 24, 2026 20:33
@amazeAmazing
amazeAmazing force-pushed the fix/dry-run-mlflow-instance-type branch 2 times, most recently from 091ed6e to eeba7e8 Compare July 27, 2026 20:56
When dry_run=True, _resolve_mlflow_resource_arn now lists existing apps but
skips creation and waiting. All trainers (SFT, DPO, RLVR, RLAIF, MultiTurnRL)
forward dry_run to MLflow resolution. 9 unit tests.
@amazeAmazing
amazeAmazing force-pushed the fix/dry-run-mlflow-instance-type branch from eeba7e8 to 7077a89 Compare July 27, 2026 21:31
@amazeAmazing amazeAmazing changed the title fix(dry_run): skip MLflow app creation and warn on unsupported instance types fix(dry_run): skip MLflow app creation and fix evaluator base_job_name Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants