docs(training-guides): daily fine-tuning pipeline with MLflow + TrustyAI

[typhoonzero]

Summary

Adds docs/en/training_guides/fine-tuning-pipeline-with-mlflow-trustyai.mdx — an end-to-end solution guide that stitches Kubeflow Pipelines, MLflow tracking + Model Registry, KServe, and TrustyAI LM-Eval into one recipe:

• SFT Qwen/Qwen3-0.6B with the HF Trainer + report_to="mlflow", register the fine-tuned model in the MLflow Model Registry.
• Deploy the registered model as a temporary KServe InferenceService.
• Evaluate it with a TrustyAI LMEvalJob and log the metrics back into the same MLflow experiment as a nested run + a tag on the model version.
• Clean up the serving resources under dsl.ExitHandler.
• Schedule the whole thing as a KFP Recurring Run.
• Compare day-over-day evaluation runs in MLflow, with a mlflow.search_runs snippet for a CI gate and a set_registered_model_alias("champion", …) snippet for promotion.

Also wires the new page into the training_guides Pick-a-path index and adds a "See also" line from pipelines-mlflow-integration.mdx.

What's been validated

TrustyAI-slice smoke on the x86 dev cluster (only TrustyAI was installed; KFP + MLflow are not present, so the KFP orchestration is not exercised):

• ModelScope download of Qwen3-0.6B onto a ceph-rbd PVC.
• KServe InferenceService on the HAMi vGPU with the platform's aml-vllm-0.20.2-cuda-13.0 runtime; /v1/completions returns the expected Qwen3 answer.
• TrustyAI LMEvalJob CRD accepts the exact spec the guide's evaluate component builds and launches python -m lm_eval with the right model_args.

Three real gotchas surfaced by the smoke are now folded into the guide (see Prereqs + Troubleshooting):

1. ta-lmes-job runs as UID 65532, so spec.pod.securityContext.fsGroup: 65532 is required or the driver dies with open …/output/stdout.log: permission denied.
2. state == "Complete" is reached on both success and failure — the guide's wait loop now also checks status.reason == "Succeeded".
3. tokenized_requests: "False" does not skip AutoTokenizer.from_pretrained(...), so air-gapped clusters must switch to the offline PVC path documented in trustyai/lm-eval.mdx.

Test plan

• yarn lint clean (verified locally).
• On a cluster with KFP + MLflow + TrustyAI + KServe present, compile and submit the pipeline from the guide and confirm parent + train + eval runs appear in the MLflow experiment and a new version is registered in the model registry.
• Attach the pipeline to a Recurring Run with 0 2 * * * and confirm two daily runs land as separate rows in the MLflow experiment.

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying alauda-ai with    Cloudflare Pages

Latest commit:	c7b63c2
Status:	✅  Deploy successful!
Preview URL:	https://91800880.alauda-ai.pages.dev
Branch Preview URL:	https://docs-finetune-pipeline-mlflo.alauda-ai.pages.dev

View logs

[typhoonzero]

Full E2E now green on x86 dev cluster (DSPO KFP + MLflow Cluster Plugin + KServe/HAMi)

Pushed 8 more troubleshooting rows in c7b63c2 covering the pod-security-context, PVC ownership, cache-directory, torch-CPU-wheel, ExitHandler, chmod-model-dir, MLflow-experiment-context and MLflow-artifact-store issues that a fresh pipeline hits on this backend.

MLflow-side proof:

=> pipeline-86afe042-...  FINISHED  tags: pvc_path=..., final_loss=3.4750, eval_acc=1.0000
=> train-86afe042-...     FINISHED  metrics: loss=3.475, perplexity=32.30
=> eval-86afe042-...      FINISHED  metrics: smoke_accuracy=1.0, smoke_hits=5

Model output was faithful to the doc's flow — the fine-tuned Qwen3-0.6B answered:

• The capital of France is → Paris. The capital of Italy is Rome…
• The largest planet in our solar system is → Jupiter
• Water at sea level boils at → 100°C
• The chemical symbol for gold is → Au
• The number of continents on Earth is → 7

The KFP workflow finished at Succeeded::9/9 end-to-end (fine-tune → deploy → evaluate → cleanup).

🤖 Generated with Claude Code