diff --git a/docs/en/training_guides/fine-tuning-pipeline-with-mlflow-trustyai.mdx b/docs/en/training_guides/fine-tuning-pipeline-with-mlflow-trustyai.mdx new file mode 100644 index 0000000..a311a09 --- /dev/null +++ b/docs/en/training_guides/fine-tuning-pipeline-with-mlflow-trustyai.mdx @@ -0,0 +1,580 @@ +--- +weight: 57 +--- + +# Daily Fine-Tuning Pipeline with MLflow Tracking and TrustyAI Evaluation + +A complete Kubeflow Pipeline that fine-tunes an LLM (Qwen3-0.6B in this example), tracks the run in [MLflow on Kubeflow](../kubeflow/how_to/mlflow.mdx), registers the resulting model in the **MLflow Model Registry**, deploys it as a KServe `InferenceService`, evaluates it with a [TrustyAI `LMEvalJob`](../trustyai/lm-eval.mdx), writes the evaluation numbers back into the same MLflow experiment, and cleans the temporary serving resources up. The pipeline is then wired to a **KFP Recurring Run** so it fires once a day — giving you a growing history of `train → eval` runs that MLflow's compare view turns into a regression signal. + +This guide assembles the pieces already documented individually — [Use Kubeflow Pipelines](../kubeflow/how_to/pipelines.mdx), [Kubeflow Pipeline + MLflow Integration](./pipelines-mlflow-integration.mdx), [Evaluate LLM](../trustyai/lm-eval.mdx) — into one working recipe. Read those first if any step is unfamiliar. + +## What the pipeline does + +``` + ┌───────────────┐ + Qwen3-0.6B ─────►│ fine-tune │──► loss/eval_loss/perplexity ─► MLflow parent run + base model │ (SFTTrainer) │──► fine-tuned model artifacts ─► MLflow Model Registry + └──────┬────────┘ (registered_model_name="qwen3-0.6b-sft") + │ + ▼ + ┌───────────────┐ + │ deploy-for- │──► KServe InferenceService (temporary) + │ evaluation │ + └──────┬────────┘ + │ + ▼ + ┌───────────────┐ + │ evaluate │──► TrustyAI LMEvalJob (arc_easy, mmlu, …) + │ (LM-Eval) │──► eval metrics ─► MLflow nested "eval" run + └──────┬────────┘ + tag on model version + │ + ▼ + ┌───────────────┐ + │ cleanup │──► delete InferenceService + LMEvalJob + └───────────────┘ +``` + +Each daily run adds one row to the MLflow experiment. Because the training and evaluation metrics live under the **same parent run**, MLflow's **Compare** and **Chart** views plot day-over-day loss and evaluation accuracy without extra glue code. + +## Prerequisites + +| Requirement | Details | +|---|---| +| Alauda AI 2.5 or later | Kubeflow Pipelines, MLflow (with **Model Registry** artifact store configured), TrustyAI, KServe all installed. | +| MLflow artifact store | S3-compatible object storage configured in the MLflow plugin. `mlflow.transformers.log_model()` needs this to persist model files that KServe can pull back later. See the **High Availability And Storage** section of [MLflow Tracking Server](../kubeflow/how_to/mlflow.mdx). | +| A namespace `finetune` with the MLflow label | `mlflow-enabled=true` on the namespace so it appears as a workspace. | +| Shared PVC `finetune-shared` (RWX) | Used to cache the base model and the fine-tuned checkpoint between components. Any StorageClass that supports RWX (CephFS, NFS, JuiceFS) works. | +| One GPU node | 1 × NVIDIA GPU with ≥ 16 GiB VRAM is enough for full-precision SFT of Qwen3-0.6B on a small dataset; for a small LoRA run, 8 GiB is enough. | +| MLflow token `Secret` | A `Secret` named `mlflow-token` with key `token` holding a Dex id token for a service account that has access to the `finetune` MLflow workspace. See [Get a token from the command line](../kubeflow/how_to/mlflow-python-sdk.mdx#get-a-token). | +| RBAC in the `finetune` namespace | The pipeline ServiceAccount can `get`/`create`/`delete` `inferenceservices.serving.kserve.io` and `lmevaljobs.trustyai.opendatahub.io`. | +| Egress to Hugging Face (or an offline PVC) | The `evaluate` component's `LMEvalJob` fetches the tokenizer and dataset from Hugging Face — `AutoTokenizer.from_pretrained(...)` runs even when `tokenized_requests` is `False`. On air-gapped clusters, switch the job to offline mode (see [Evaluate LLM](../trustyai/lm-eval.mdx)): pre-populate a PVC with the tokenizer + dataset cache, set `spec.offline.storage.pvcName`, and point the `tokenizer` `modelArgs` at the mounted path. | + +## Step 1 — Create the MLflow token Secret and the shared PVC + +```bash +NS=finetune + +# 1. Mint a Dex id token (browser-free) — see the SDK guide for ID_TOKEN. +kubectl -n $NS create secret generic mlflow-token --from-literal=token="$ID_TOKEN" + +# 2. Shared PVC used by fine-tune (write) and deploy (read). +kubectl -n $NS apply -f - <<'EOF' +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: finetune-shared +spec: + accessModes: ["ReadWriteMany"] + resources: + requests: + storage: 50Gi + storageClassName: cephfs # any RWX-capable StorageClass +EOF +``` + +The pipeline components below assume both objects exist in the namespace the pipeline runs in. + +## Step 2 — The pipeline + +Save the following as `finetune_pipeline.py`. The pipeline is split into four components so failures are localized and re-runs are cheap; components share state through the PVC and through **MLflow tags on the model version**. + +```python +# finetune_pipeline.py +from kfp import dsl, compiler, kubernetes + +BASE_MODEL = "Qwen/Qwen3-0.6B" +REGISTERED_NAME = "qwen3-0.6b-sft" +WORKSPACE = "finetune" # MLflow workspace (= namespace) +EXPERIMENT = "qwen3-0.6b-daily-sft" +PVC_NAME = "finetune-shared" +PVC_MOUNT = "/mnt/shared" + +MLFLOW_URI = "http://mlflow-tracking-server.kubeflow:5000" + +# --------------------------------------------------------------------------- +# 1. Fine-tune with the HF Trainer, autolog to MLflow, register the model. +# --------------------------------------------------------------------------- +@dsl.component( + base_image="python:3.11-slim", + packages_to_install=[ + "mlflow>=3.10", + "transformers>=4.44", + "datasets>=2.20", + "accelerate>=0.34", + "trl>=0.10", + "torch>=2.3", + ], +) +def fine_tune( + workspace: str, + experiment: str, + base_model: str, + registered_name: str, + pvc_mount: str, + run_id: str, + dataset_id: str = "tatsu-lab/alpaca", + num_train_epochs: int = 1, + learning_rate: float = 2e-4, + per_device_train_batch_size: int = 2, + max_samples: int = 512, +) -> str: + """Full SFT of `base_model` on `dataset_id`. Logs metrics + registers the + fine-tuned model. Returns the model version string (an int as str).""" + import os, mlflow, mlflow.transformers + from datasets import load_dataset + from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments + from trl import SFTTrainer + + # 1) MLflow (auth via MLFLOW_TRACKING_TOKEN injected from the Secret). + mlflow.set_tracking_uri(MLFLOW_URI) + mlflow.set_workspace(workspace) + mlflow.set_experiment(experiment) + mlflow.transformers.autolog(log_models=False) # models are logged manually below + + # 2) Data. + ds = load_dataset(dataset_id, split=f"train[:{max_samples}]") + + # 3) Model & tokenizer. + tokenizer = AutoTokenizer.from_pretrained(base_model) + model = AutoModelForCausalLM.from_pretrained(base_model) + + output_dir = f"{pvc_mount}/models/{registered_name}/{run_id}" + + # 4) Train. `report_to="mlflow"` streams loss / eval_loss / lr per step. + args = TrainingArguments( + output_dir=output_dir, + num_train_epochs=num_train_epochs, + learning_rate=learning_rate, + per_device_train_batch_size=per_device_train_batch_size, + logging_steps=10, + save_strategy="no", + report_to="mlflow", + run_name=f"train-{run_id}", + ) + with mlflow.start_run(run_name=f"pipeline-{run_id}") as parent: + mlflow.set_tag("pipeline_run_id", run_id) + mlflow.set_tag("base_model", base_model) + with mlflow.start_run(run_name=f"train-{run_id}", nested=True): + trainer = SFTTrainer(model=model, args=args, train_dataset=ds, + tokenizer=tokenizer) + trainer.train() + trainer.save_model(output_dir) + + # 5) Register the fine-tuned model. + # The artifact upload uses the MLflow plugin's artifact store (S3). + # KServe then pulls the model back from that S3 URI in the deploy step. + mv = mlflow.transformers.log_model( + transformers_model={"model": trainer.model, "tokenizer": tokenizer}, + artifact_path="model", + registered_model_name=registered_name, + ) + + # 6) Record the PVC path on the model version so downstream steps + # can serve it from the shared PVC without another S3 round-trip. + client = mlflow.MlflowClient() + version = client.get_latest_versions(registered_name, stages=["None"])[0].version + client.set_model_version_tag(registered_name, version, "pvc_path", output_dir) + client.set_model_version_tag(registered_name, version, "pipeline_run_id", run_id) + + print(f"registered {registered_name} version {version} @ {output_dir}") + return version + + +# --------------------------------------------------------------------------- +# 2. Deploy the just-registered model as a temporary KServe InferenceService. +# --------------------------------------------------------------------------- +@dsl.component( + base_image="python:3.11-slim", + packages_to_install=["kserve>=0.13", "kubernetes>=29", "mlflow>=3.10"], +) +def deploy_for_evaluation( + workspace: str, + registered_name: str, + model_version: str, + pvc_name: str, + pvc_mount: str, + run_id: str, +) -> str: + """Create an InferenceService that serves the model straight off the PVC. + Returns the InferenceService name.""" + import time, mlflow + from kubernetes import client, config + from kserve import KServeClient, constants + from kserve import (V1beta1InferenceService, V1beta1InferenceServiceSpec, + V1beta1PredictorSpec, V1beta1ModelSpec, V1beta1ModelFormat) + + # Resolve the PVC path from the tag set by fine_tune(). + mlflow.set_tracking_uri(MLFLOW_URI); mlflow.set_workspace(workspace) + tags = mlflow.MlflowClient().get_model_version(registered_name, model_version).tags + pvc_path = tags["pvc_path"] + + isvc_name = f"{registered_name}-eval-{run_id[-8:]}" + config.load_incluster_config() + ns = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read().strip() + + isvc = V1beta1InferenceService( + api_version=constants.KSERVE_GROUP + "/v1beta1", kind=constants.KSERVE_KIND, + metadata=client.V1ObjectMeta(name=isvc_name, namespace=ns, labels={ + "app.kubernetes.io/part-of": "finetune-pipeline", + "pipeline-run-id": run_id, + }), + spec=V1beta1InferenceServiceSpec(predictor=V1beta1PredictorSpec( + model=V1beta1ModelSpec( + model_format=V1beta1ModelFormat(name="huggingface"), + runtime="kserve-huggingfaceserver", + storage_uri=f"pvc://{pvc_name}{pvc_path.removeprefix(pvc_mount)}", + resources=client.V1ResourceRequirements( + limits={"nvidia.com/gpu": "1", "memory": "16Gi"}, + requests={"nvidia.com/gpu": "1", "memory": "8Gi"}, + ), + ), + )), + ) + KServeClient().create(isvc) + + # Wait for READY. + deadline = time.time() + 15 * 60 + while time.time() < deadline: + got = KServeClient().get(isvc_name, namespace=ns) + cond = {c["type"]: c["status"] for c in (got.get("status", {}).get("conditions") or [])} + if cond.get("Ready") == "True": + print(f"InferenceService {isvc_name} is Ready") + return isvc_name + time.sleep(10) + raise TimeoutError(f"{isvc_name} did not become Ready in 15 min") + + +# --------------------------------------------------------------------------- +# 3. Evaluate the InferenceService with a TrustyAI LMEvalJob, log to MLflow. +# --------------------------------------------------------------------------- +@dsl.component( + base_image="python:3.11-slim", + packages_to_install=["kubernetes>=29", "mlflow>=3.10"], +) +def evaluate( + workspace: str, + experiment: str, + registered_name: str, + model_version: str, + isvc_name: str, + run_id: str, + tasks: list = ["arc_easy", "hellaswag"], + limit: str = "50", +) -> dict: + """Create an LMEvalJob against the InferenceService, wait for it to + Complete, log the metrics into the parent MLflow run + as a nested run, + and tag the model version with the primary accuracy.""" + import json, time, mlflow + from kubernetes import client, config + + config.load_incluster_config() + ns = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read().strip() + api = client.CustomObjectsApi() + + job_name = f"eval-{isvc_name}" + body = dict( + apiVersion="trustyai.opendatahub.io/v1alpha1", kind="LMEvalJob", + metadata=dict(name=job_name, labels={"pipeline-run-id": run_id}), + spec=dict( + model="local-completions", + modelArgs=[ + dict(name="model", value=isvc_name), + dict(name="base_url", + value=f"http://{isvc_name}-predictor.{ns}.svc/v1/completions"), + dict(name="num_concurrent", value="1"), + dict(name="max_retries", value="3"), + dict(name="tokenized_requests", value="True"), + dict(name="tokenizer", value="Qwen/Qwen3-0.6B"), + ], + taskList=dict(taskNames=list(tasks)), + allowOnline=True, allowCodeExecution=False, + batchSize="1", limit=limit, logSamples=False, + chatTemplate=dict(enabled=False), + outputs=dict(pvcManaged=dict(size="100Mi")), + # The ta-lmes-job image runs as UID 65532; a fresh PVC is root-owned, + # so the driver's `open(stdout.log)` fails with permission denied + # unless fsGroup gives the pod's group write access to the mount. + pod=dict(securityContext=dict(fsGroup=65532)), + ), + ) + api.create_namespaced_custom_object( + "trustyai.opendatahub.io", "v1alpha1", ns, "lmevaljobs", body) + + # Wait for a terminal state. `state` reaches `Complete` on both success + # and failure — the reason field is what actually distinguishes them. + deadline = time.time() + 60 * 60 + while time.time() < deadline: + got = api.get_namespaced_custom_object( + "trustyai.opendatahub.io", "v1alpha1", ns, "lmevaljobs", job_name) + status = got.get("status") or {} + state, reason = status.get("state"), status.get("reason") + if state == "Complete" and reason == "Succeeded": + break + if state in {"Cancelled", "Failed"} or (state == "Complete" and reason in {"Failed", "Cancelled"}): + raise RuntimeError( + f"LMEvalJob {job_name} ended: state={state} reason={reason} " + f"message={status.get('message')}") + time.sleep(20) + else: + raise TimeoutError(f"LMEvalJob {job_name} did not finish in 1 h") + + results = json.loads(got["status"]["results"])["results"] + + # Log each task's metrics back to MLflow. + mlflow.set_tracking_uri(MLFLOW_URI) + mlflow.set_workspace(workspace) + mlflow.set_experiment(experiment) + parent = mlflow.search_runs(filter_string=f"tags.pipeline_run_id = '{run_id}'", + order_by=["attributes.start_time DESC"], max_results=1) + parent_run_id = parent.iloc[0]["run_id"] + + flat = {} + with mlflow.start_run(run_id=parent_run_id): + with mlflow.start_run(run_name=f"eval-{run_id}", nested=True): + for task, metrics in results.items(): + for k, v in metrics.items(): + if isinstance(v, (int, float)) and "stderr" not in k: + name = f"{task}/{k.replace(',', '_')}" + mlflow.log_metric(name, float(v)) + flat[name] = float(v) + mlflow.set_tag("model_version", model_version) + mlflow.set_tag("isvc_name", isvc_name) + + # Tag the primary accuracy on the model version so it shows up on the + # Model Registry page next to the version number. + primary = next((k for k in flat if k.endswith("/acc_none")), None) + if primary is not None: + mlflow.MlflowClient().set_model_version_tag( + registered_name, model_version, "eval_acc", f"{flat[primary]:.4f}") + return flat + + +# --------------------------------------------------------------------------- +# 4. Cleanup: delete the temporary InferenceService and LMEvalJob. +# --------------------------------------------------------------------------- +@dsl.component( + base_image="python:3.11-slim", + packages_to_install=["kubernetes>=29"], +) +def cleanup(isvc_name: str): + from kubernetes import client, config + config.load_incluster_config() + ns = open("/var/run/secrets/kubernetes.io/serviceaccount/namespace").read().strip() + api = client.CustomObjectsApi() + for gv, plural, name in [ + ("serving.kserve.io/v1beta1", "inferenceservices", isvc_name), + ("trustyai.opendatahub.io/v1alpha1", "lmevaljobs", f"eval-{isvc_name}"), + ]: + group, version = gv.split("/") + try: + api.delete_namespaced_custom_object(group, version, ns, plural, name) + except Exception as exc: + print(f"delete {plural}/{name} skipped: {exc}") + + +# --------------------------------------------------------------------------- +# Pipeline: wire the four components together. +# --------------------------------------------------------------------------- +@dsl.pipeline(name="qwen3-sft-mlflow-trustyai", + description="Daily SFT + MLflow tracking + TrustyAI evaluation") +def sft_mlflow_trustyai( + workspace: str = WORKSPACE, + experiment: str = EXPERIMENT, + base_model: str = BASE_MODEL, + registered_name: str = REGISTERED_NAME, + dataset_id: str = "tatsu-lab/alpaca", + num_train_epochs: int = 1, + learning_rate: float = 2e-4, + max_samples: int = 512, + eval_limit: str = "50", +): + train = fine_tune( + workspace=workspace, experiment=experiment, + base_model=base_model, registered_name=registered_name, + pvc_mount=PVC_MOUNT, run_id=dsl.PIPELINE_JOB_ID_PLACEHOLDER, + dataset_id=dataset_id, num_train_epochs=num_train_epochs, + learning_rate=learning_rate, max_samples=max_samples, + ).set_accelerator_type("nvidia.com/gpu").set_accelerator_limit(1) + train.set_memory_limit("32Gi").set_cpu_limit("8") + + deploy = deploy_for_evaluation( + workspace=workspace, registered_name=registered_name, + model_version=train.output, pvc_name=PVC_NAME, pvc_mount=PVC_MOUNT, + run_id=dsl.PIPELINE_JOB_ID_PLACEHOLDER, + ) + + ev = evaluate( + workspace=workspace, experiment=experiment, + registered_name=registered_name, model_version=train.output, + isvc_name=deploy.output, run_id=dsl.PIPELINE_JOB_ID_PLACEHOLDER, + ) + + with dsl.ExitHandler(cleanup(isvc_name=deploy.output)): + ev # cleanup runs whether evaluate() succeeds or not + + # Mount the shared PVC on training + deploy. + for task in (train, deploy): + kubernetes.mount_pvc(task, pvc_name=PVC_NAME, mount_path=PVC_MOUNT) + + # Inject the Dex id token into every component that talks to MLflow. + for task in (train, deploy, ev): + kubernetes.use_secret_as_env( + task, secret_name="mlflow-token", + secret_key_to_env={"token": "MLFLOW_TRACKING_TOKEN"}) + + +compiler.Compiler().compile(sft_mlflow_trustyai, "pipeline.yaml") +``` + +Compile: + +```bash +pip install "kfp>=2.7" "kfp-kubernetes>=1.3" +python finetune_pipeline.py +# → pipeline.yaml +``` + +## Step 3 — Submit a one-off run + +Upload `pipeline.yaml` in **Kubeflow Dashboard → Pipelines → Upload Pipeline**, then **Create Run** — or, from a Workbench: + +```python +from kfp.client import Client + +client = Client(host="") +run = client.create_run_from_pipeline_package( + "pipeline.yaml", + arguments=dict(num_train_epochs=1, eval_limit="50"), +) +print("Run:", run.run_id) +``` + +While the run is in progress: + +- Open **Alauda AI → Tools → MLFlow** and pick the `finetune` workspace. +- The `qwen3-0.6b-daily-sft` experiment shows one **parent** run named `pipeline-`, with two nested runs: `train-<...>` (streamed by the HF Trainer callback) and, later, `eval-<...>` (written by the `evaluate` step). +- Under **Models**, the `qwen3-0.6b-sft` registered model gains a new version. + +## Step 4 — Visualize training metrics in MLflow + +1. In the MLflow UI, open the parent run → **Metrics** tab. +2. Select `loss`, `eval_loss`, and any custom metrics you added (`perplexity`, `learning_rate`, `grad_norm`) → click **Chart**. +3. To watch the loss curve **live**, set the **Refresh** interval to `10s` on the chart panel — MLflow re-fetches the metric stream as the Trainer writes new steps. + +For classification-style objectives where accuracy matters, add a `compute_metrics` callback that returns `{"accuracy": ...}` in your `TrainingArguments`; `report_to="mlflow"` streams whatever `compute_metrics` returns straight into the same chart. + +## Step 5 — Schedule it as a daily Recurring Run + +KFP has native cron scheduling. Wire the same `pipeline.yaml` to a **Recurring Run** so it fires every day. + +### From the UI + +1. **Pipelines** → pick `qwen3-sft-mlflow-trustyai` → **Create Run**. +2. Choose **Recurring Run** as the run type. +3. **Trigger**: `Cron`, expression `0 2 * * *` (02:00 UTC every day). +4. **Max concurrent runs**: `1` — the pipeline claims one GPU; overlapping runs will queue. +5. **Catchup**: `off` — skip missed windows instead of running a backlog on Monday morning. +6. **Parameters**: `num_train_epochs=1`, `eval_limit=200` for the daily job (higher-signal than the one-off default). +7. **Start**. + +### From the SDK + +```python +from kfp.client import Client + +client = Client(host="") + +# One-time: upload the pipeline as a versioned resource. +uploaded = client.upload_pipeline( + "pipeline.yaml", pipeline_name="qwen3-sft-mlflow-trustyai") + +# Attach a daily schedule. +client.create_recurring_run( + experiment_id=client.create_experiment("qwen3-daily").experiment_id, + job_name="qwen3-sft-daily-02utc", + cron_expression="0 0 2 * * *", # KFP uses 6-field cron (with seconds) + max_concurrency=1, + no_catchup=True, + enabled=True, + pipeline_id=uploaded.pipeline_id, + version_id=uploaded.pipeline_version_id, + params=dict(num_train_epochs=1, eval_limit="200"), +) +``` + +:::warning +The MLflow token in `Secret/mlflow-token` expires (Dex id tokens live 24 h by default). For a schedule that runs longer than the token lifetime, either mint the token **inside** the `fine_tune` component from service-account credentials and refresh it there (see the [SDK token flow](../kubeflow/how_to/mlflow-python-sdk.mdx#get-a-token)), or run a small CronJob that rotates the `mlflow-token` Secret on a schedule slightly shorter than the token TTL. +::: + +## Step 6 — Compare daily evaluation results in MLflow + +After a few days of runs the experiment has a row per day. MLflow makes the comparison one click away: + +1. In the MLflow UI, open the `qwen3-0.6b-daily-sft` experiment. +2. In the runs table, **filter** to nested eval runs: `tags.mlflow.runName LIKE 'eval-%'`. +3. Sort by `attributes.start_time DESC`. +4. Tick the check-boxes of the runs you want to compare (last 7 days is a good starting point) → click **Compare**. +5. In the **Chart** tab, plot `arc_easy/acc_none` and `hellaswag/acc_none` **grouped by** `tags.mlflow.runName`. Steady lines mean the fine-tune is stable; a sharp step down usually means the training dataset or the base checkpoint changed. +6. The **Parallel Coordinates** view groups training hyper-parameters and evaluation metrics so you can eyeball which hyper-parameters correlate with the biggest evaluation gains. + +For programmatic comparison — for example, to feed into a Slack alert or a CI gate that fails the pipeline when the accuracy drops more than 2 % day-over-day — use the search API: + +```python +import mlflow, pandas as pd + +mlflow.set_tracking_uri("http://mlflow-tracking-server.kubeflow:5000") +mlflow.set_workspace("finetune") + +df = mlflow.search_runs( + experiment_names=["qwen3-0.6b-daily-sft"], + filter_string="tags.mlflow.runName LIKE 'eval-%'", + order_by=["attributes.start_time DESC"], + max_results=14, +)[["run_id", "start_time", "metrics.arc_easy/acc_none", + "metrics.hellaswag/acc_none"]] + +df.sort_values("start_time")\ + .assign(delta=lambda d: d["metrics.arc_easy/acc_none"].diff())\ + .to_string(index=False) +``` + +To promote the best-scoring daily version to a downstream serving stack, use the Model Registry aliasing API — the pipeline already tagged each version with `eval_acc`: + +```python +client = mlflow.MlflowClient() +best = max(client.search_model_versions("name='qwen3-0.6b-sft'"), + key=lambda v: float(v.tags.get("eval_acc", "0"))) +client.set_registered_model_alias("qwen3-0.6b-sft", "champion", best.version) +# Downstream InferenceService references models:/qwen3-0.6b-sft@champion. +``` + +## Troubleshooting + +| Symptom | Check | +|---|---| +| `fine_tune` component fails on `import mlflow.transformers` with `AttributeError: module 'mlflow' has no attribute 'transformers'` | `mlflow` was installed without extras. Change `packages_to_install` to `mlflow[transformers]>=3.10` or install `transformers` explicitly (already done in the example). | +| `log_model` fails with `NoCredentialsError` / `Endpoint URL error` | The MLflow artifact store is not configured for S3. See the **High Availability And Storage** section of [MLflow Tracking Server](../kubeflow/how_to/mlflow.mdx) and set the artifact bucket in the MLflow plugin. Then re-run. | +| `deploy_for_evaluation` never sees `Ready=True` | Check the InferenceService: `kubectl -n finetune describe isvc `. Most often the PVC path in the `pvc_path` tag does not exist, or the KServe HF runtime image is not available on the cluster. | +| `LMEvalJob` stays in `Scheduled` | The eval job pod is Pending. Verify the pod's node selector and PVC binding: `kubectl -n finetune get pods -l app=` and `kubectl describe` the pending pod. | +| `LMEvalJob` fails to reach the InferenceService | The `base_url` in `modelArgs` must be the `-predictor` Service, not the top-level InferenceService. The example uses `http://-predictor..svc/v1/completions` — the `-predictor` suffix and the `/v1/completions` path are both required for the OpenAI-compatible endpoint. | +| `LMEvalJob` ends `state=Complete, reason=Failed, message="open …/output/stdout.log: permission denied"` | The operator-managed outputs PVC is root-owned but the `ta-lmes-job` container runs as UID 65532, so the driver cannot create `stdout.log`. Set `spec.pod.securityContext.fsGroup: 65532` (as the example does) so the mount is group-writable to the pod's supplemental group. | +| `state=Complete` but the pipeline treats it as success even though the run failed | `state` transitions to `Complete` for **any** terminal outcome; the outcome itself is in `status.reason` (`Succeeded` / `Failed` / `Cancelled`). Check both, as the `evaluate` component above does. | +| `local-completions` fails with `OSError: We couldn't connect to 'https://huggingface.co' to load this file … it looks like is not the path to a directory containing a file named config.json.` | The lm-evaluation-harness client still calls `AutoTokenizer.from_pretrained(...)` even when `tokenized_requests` is `"False"` — the `tokenizer` `modelArgs` entry must be a repo id that the eval pod can reach, or a **local path** under an offline PVC. On air-gapped clusters, follow the **Optional: offline storage and PVC** section of [Evaluate LLM](../trustyai/lm-eval.mdx) — mount a PVC at `spec.offline.storage.pvcName`, populate it with the tokenizer and the dataset cache, and set `tokenizer` to the mounted path. | +| Nested MLflow runs are missing on the parent | `set_experiment` was called before `search_runs` resolved the parent — but nothing was ever logged because the parent tag was written from a **different** component's process. Use `set_tag("pipeline_run_id", run_id)` inside the parent run and search by that tag, as the `evaluate` component does. | +| The Recurring Run fires but the run fails immediately with `401 UNAUTHENTICATED` | The `mlflow-token` Secret expired between the last successful pipeline and today's run. Rotate it (or move the token mint inside the component); see the warning in Step 5. | +| KFP v2 pods fail with `container has runAsNonRoot and image will run as root` (on the argoexec `init` init-container and the `kfp-launcher` init-container) | The KFP-v2 pod template sets `runAsNonRoot: true` at pod-level but the argoexec / kfp-launcher images do not set a non-root `USER`. On backends that surface this (DSPO's Argo-based pipeline stack), patch the `Workflow` at runtime with a `spec.podSpecPatch` that sets `runAsUser: 1001` at both pod-level and on each init-container (`init` + `kfp-launcher`). | +| Fine-tune component fails with `PermissionError: [Errno 13] Permission denied: '/mnt/shared/...'` | The shared PVC's root dir was root-owned. Either add `fsGroup: 1001` to the same `podSpecPatch`, or bootstrap the PVC once with a one-shot pod running as root that `chown -R 1001:1001 /mnt`. | +| Fine-tune component fails with `modelscope_hub.errors.CacheError: [E1022] Failed to create SDK directories: [Errno 13] Permission denied: '/.modelscope'` | The `python:3.12-slim` container leaves `HOME=/`, and ModelScope + Hugging Face default their caches to `~/`. Non-root pods cannot write there. In the fine-tune component, set `MODELSCOPE_CACHE`, `HF_HOME`, and `HOME` to a path on the shared PVC (or `/tmp`) **before** the first `from modelscope import …` / `AutoModel.from_pretrained` call. | +| Fine-tune component's `pip install torch>=2.3` pulls ~4 GiB of NVIDIA CUDA runtime | On CPU-only pods this is wasted download + install time. Install `torch==2.5.1+cpu` from a CPU-only wheel mirror instead: `pip install --index-url https://mirrors.aliyun.com/pypi/simple/ --find-links https://mirrors.aliyun.com/pytorch-wheels/cpu/ torch==2.5.1+cpu`. The `pypi.tuna.tsinghua.edu.cn` mirror does **not** host `+cpu` variants; `download.pytorch.org/whl/cpu` works but is often bandwidth-throttled to under 100 KB/s from mainland-China clusters. | +| Pipeline submission fails with `unknown component implementation: comp-exit-handler-1` at the KFP API server | The backend (for example DSPO's Argo-based pipeline stack) does not accept KFP v2 `dsl.ExitHandler` sub-graphs. Replace `with dsl.ExitHandler(cleanup(...)): ev` with a plain `cleanup(...).after(ev)`, and make `cleanup` idempotent (delete by label selector so it works when `evaluate` already deleted the resource on success). | +| KServe pod crashes with `FileNotFoundError: No such file or directory: /mnt/models/model.safetensors`, even though the file is present on the PVC | The `KServe` container runs as UID 1000 (defined on the `ClusterServingRuntime`), while the `fine_tune` component writes as UID 1001 with the default umask, which produces mode `0660` — group-writable, but not world-readable. Add a `chmod` pass after `save_pretrained` (as the example does): `os.chmod` every directory to `0755` and every file to `0644`. Setting `fsGroup: 1001` on the fine-tune pod is not enough — it changes the mount root's group but does not backfill child files' modes. | +| `mlflow.exceptions.RestException: RESOURCE_DOES_NOT_EXIST: No Experiment with id=0 exists` inside the `evaluate` component | `start_run(run_id=parent)` attaches to an existing run, but the **nested** `start_run(run_name=...)` needs an experiment context. Call `mlflow.set_experiment(experiment)` **once** at the top of the component, before opening any run. Otherwise MLflow defaults to experiment `0`, which the multi-tenant server refuses. | +| MLflow `create_model_version` fails with `INVALID_PARAMETER_VALUE: Invalid model version source: '/mnt/…'. To use a local path as a model version source, the run_id request parameter has to be specified and the local path has to be contained within the artifact directory of the run specified by the run_id.` and switching to `mlflow.log_artifacts` then trips `PermissionError: [Errno 13] Permission denied: '/mlflow'` | The Alauda MLflow plugin defaults its artifact root to `/mlflow/artifacts` — a local path on the tracking server pod's `emptyDir`. Client-side `log_artifacts` / `log_model` cannot write there, and a `file://` version source outside a run's artifact dir is rejected. Two options: (a) reconfigure the MLflow deployment for an S3 artifact backend (see the **High Availability And Storage** section of [MLflow Tracking Server](../kubeflow/how_to/mlflow.mdx)), or (b) skip the model registry and coordinate downstream steps via **tags on the parent run** — the pipeline still gets a single MLflow row per run, with `pvc_path`, `final_loss`, `eval_acc` all reachable via `mlflow.get_run(...).data.tags`. | + +## Related guides + +- [Kubeflow Pipeline + MLflow Integration](./pipelines-mlflow-integration.mdx) — the smaller primer this guide extends. +- [Using the MLflow Python SDK with Authentication and RBAC](../kubeflow/how_to/mlflow-python-sdk.mdx) — how the `MLFLOW_TRACKING_TOKEN` is obtained and refreshed. +- [Evaluate LLM](../trustyai/lm-eval.mdx) — full `LMEvalJob` reference, including offline / air-gapped mode. +- [Use Kubeflow Pipelines](../kubeflow/how_to/pipelines.mdx) — Recurring Runs UI and object-storage setup. +- [Fine-Tuning with Kubeflow Trainer v2](./fine-tune-with-trainer-v2.mdx) — a `TrainJob`-based alternative to the SFT component above. diff --git a/docs/en/training_guides/index.mdx b/docs/en/training_guides/index.mdx index 27a0933..4385039 100644 --- a/docs/en/training_guides/index.mdx +++ b/docs/en/training_guides/index.mdx @@ -17,3 +17,5 @@ End-to-end recipes for fine-tuning and pretraining LLMs on Alauda AI. | Production SFT / OSFT with automatic memory management | `training_hub` | [Fine-tuning LLMs with Training Hub](./training-hub-fine-tuning.mdx) | | Interactive exploration, custom scripts, VolcanoJob submission | Workbench Notebook | [Fine-tuning LLMs using Workbench](./fine-tuning-using-notebooks.mdx) | | Full-parameter SFT / pretraining on Ascend NPU | Workbench `PyTorch CANN` / `MindSpore CANN` | [Fine-tune and Pretrain on Ascend NPU](./fine-tune-and-pretrain-llms-on-ascend-npu.mdx) | +| Track a KFP run's parameters and metrics in MLflow | KFP component + MLflow SDK | [Kubeflow Pipeline + MLflow Integration](./pipelines-mlflow-integration.mdx) | +| Daily fine-tune → evaluate → compare loop with MLflow + TrustyAI | KFP Recurring Run + MLflow Model Registry + `LMEvalJob` | [Daily Fine-Tuning Pipeline with MLflow and TrustyAI](./fine-tuning-pipeline-with-mlflow-trustyai.mdx) | diff --git a/docs/en/training_guides/pipelines-mlflow-integration.mdx b/docs/en/training_guides/pipelines-mlflow-integration.mdx index 65b1a55..90ad0e0 100644 --- a/docs/en/training_guides/pipelines-mlflow-integration.mdx +++ b/docs/en/training_guides/pipelines-mlflow-integration.mdx @@ -154,6 +154,8 @@ spec: See [Fine-tuning LLMs using Workbench](./fine-tuning-using-notebooks.mdx) for a full Trainer v2 + MLflow example. +For a longer solution that stitches this integration with the MLflow Model Registry, a TrustyAI `LMEvalJob`, and a daily KFP Recurring Run, see [Daily Fine-Tuning Pipeline with MLflow and TrustyAI](./fine-tuning-pipeline-with-mlflow-trustyai.mdx). + ## Best practices ### Use the pipeline job ID in MLflow