feat(serve): add --generation-config CLI for server sampling defaults

[lvhan028]

Motivation

LMDeploy's OpenAI-compatible API server previously relied on hard-coded protocol defaults for sampling parameters (e.g. temperature=0.7, top_k=40). This made it difficult to align server behavior with a model's own HuggingFace generation_config.json, which many models ship with recommended decoding settings.

vLLM exposes a similar --generation-config flag to load HF generation defaults at startup. This PR brings comparable behavior to LMDeploy:

• Load a model's HF generation_config.json as server-side defaults when users omit sampling fields in requests.
• Allow opting out via --generation-config lmdeploy to keep LMDeploy/GenerationConfig defaults.
• Support loading from a custom folder path when needed.

To support proper merge semantics, protocol sampling fields are changed to default to None (meaning "not specified by the user") instead of fixed values. Unspecified fields fall back to HF defaults (if loaded) and then to GenerationConfig dataclass defaults.

Modification

CLI

• Add --generation-config to lmdeploy serve api_server (default: auto).
  • auto: load generation_config.json from the model path via HuggingFace GenerationConfig.from_pretrained().
  • lmdeploy: do not load HF config; use LMDeploy defaults only.
  • <path>: load from a custom directory.

Core module (lmdeploy/serve/core/generation_config.py)

Introduce a small helper module

API server integration

• Parse --generation-config once at startup and store the result in VariableInterface.default_gen_config.
• Route chat completions, completions, generate, Responses API, and Anthropic endpoints through build_generation_config() for consistent merge behavior.
• Remove redundant same-name pass-through kwargs at call sites; keep only renamed (stop → stop_words, seed → random_seed), computed (logprobs), or raw-json fields (migration_request, with_cache, preserve_cache) in extra_kwargs.

Protocol changes

• Set sampling-related fields in OpenAI/Responses/Anthropic protocols to None defaults (e.g. temperature, top_p, top_k, repetition_penalty, min_p) so "user did not send" can be distinguished from an explicit value.

Behavior notes

• When --generation-config lmdeploy and the user omits sampling params, defaults come from GenerationConfig (temperature=0.8, top_k=50, etc.), not the old protocol defaults (0.7 / 40).
• max_new_tokens is not taken from HF config; it is resolved from max_completion_tokens / max_tokens on the request, with engine-level fallback when unset.
• do_sample=True is always set for serving requests.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

[grimoire]

Since ResponsesRequest uses extra='allow' at lmdeploy/serve/openai/responses/protocol.py:36, /v1/responses clients can send unsupported GenerationConfig fields like return_ppl, with_cache, migration_request, output_logits, or stop_token_ids, and they flow into the engine. I confirmed return_ppl=True, with_cache=True, stop_token_ids=[1] becomes active in GenerationConfig. Please restrict extraction to declared request fields or an endpoint-specific allowlist.

from agent.