Add GRPO Optimizer to DSPy (#8171)

* D1 for GRPO

* Improve type for arbor

* Add temp test script for grpo

* Add note about assumption of same inputs to all predictors

* Disable LM cache in GRPO

* Add support for valset

* Add configurable variable module invocation handling strategy

* Noahs dspy.LM changes and dspy.ArborProvider implementation

* Add latest arbor changes

* First working grpo version

* Add modules

* Add training args in initialize

* Fix grpo

* Add batches

* Update finetuning infra

* Revise server interface

* Update example script

* Move temporary interface to a separate file

* Add LM level reinforce interface

* Update testing script

* Update api_base access for finetune

* Style check

* Style check all

* Add Test script with MATH dataset

* Ensure grpo trainer does not crash due to format issues, but temporary fix

* Add error log

* Fix termination

* Delete temp files

* Add diff

* Add model update endpoint support

* Remove experimental flag

* Remove extra files

* Add GRPO error resiliency to avoid parsing failures lead to crashes

* Param Passthrough and Consistent Tutorial Script (#3)

* Add param passthrough and default banking77 tutorial

* Add more threads

* Update banking tutorial

---------

Co-authored-by: Noah Ziems <nziems2@nziems2@nd.edu>

* Lower beta param for banking tutorial

* Add warning on no training data

* Add train logging to GRPIO

* Add max_prompt_length and max_completion_length support

* fix litellm retries

* no jsonadapter

* fix errors

* fix tests

* fix tests

* add the retry strategy back

* Add working implementation of format errors and negative rewards

* Fix bugs in validation

* Add validation logic to grpo

* Add more supported args

* Support max grad norm

* Add Train Shuffling logic

* Add lora support

* Add soft format rewards

* Disable proivide_traceback in all grpo invoked evaluates

* Remove temporary tutorial script

* Revert classification finetuning tutorial

* Comment out json adapter test

* Fix ruff errors

* Add teacher (#8)

* Modify teacher preparation logic

* Re-add teachers to GRPO

* Style fix

* Update tutorial script

* Housekeeping

* Revert number of train steps

* Address PR comments

* Add wandb support for GRPO training runs

* Add completion logging

* Add logging steps support

* update report_to to be default none

* Add max_context_length

* Fix num_samples_per_input computation

* Checkpointing Endpoints (#10)

* Fix typo

* Fix checkpoint url

* fix merge conflict leftover

* shorten the warning message in json adapter

* fix the error piping

---------

Co-authored-by: Lakshya A Agrawal <lakshyaaagrawal@berkeley.edu>
Co-authored-by: Dilara Soylu <21346670+dilarasoylu@users.noreply.github.com>
Co-authored-by: Noah Ziems <nziems2@nziems2@nd.edu>
Co-authored-by: chenmoneygithub <chen.qian@databricks.com>

Author: 5 people

docs/docs/tutorials/classification_finetuning/index.ipynb

@@ -211,7 +211,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [

dspy/adapters/json_adapter.py

@@ -62,8 +62,8 @@ def __call__(
             structured_output_model = _get_structured_outputs_response_format(signature)
             lm_kwargs["response_format"] = structured_output_model
             return super().__call__(lm, lm_kwargs, signature, demos, inputs)
-        except Exception as e:
-            logger.warning(f"Failed to use structured output format. Falling back to JSON mode. Error: {e}")
+        except Exception:
+            logger.warning("Failed to use structured output format, falling back to JSON mode.")
             try:
                 lm_kwargs["response_format"] = {"type": "json_object"}
                 return super().__call__(lm, lm_kwargs, signature, demos, inputs)

dspy/clients/lm.py

@@ -11,7 +11,7 @@
 import dspy
 from dspy.clients.cache import request_cache
 from dspy.clients.openai import OpenAIProvider
-from dspy.clients.provider import Provider, TrainingJob
+from dspy.clients.provider import Provider, TrainingJob, ReinforceJob
 from dspy.clients.utils_finetune import TrainDataFormat
 from dspy.dsp.utils.settings import settings
 from dspy.utils.callback import BaseCallback
@@ -188,10 +188,6 @@ def finetune(
     ) -> TrainingJob:
         from dspy import settings as settings
 
-        err = "Fine-tuning is an experimental feature."
-        err += " Set `dspy.settings.experimental` to `True` to use it."
-        assert settings.experimental, err
-
         err = f"Provider {self.provider} does not support fine-tuning."
         assert self.provider.finetunable, err
 
@@ -212,6 +208,17 @@ def thread_function_wrapper():
 
         return job
 
+    def reinforce(self, train_kwargs) -> ReinforceJob:
+        # TODO(GRPO Team): Should we return an initialized job here?
+        from dspy import settings as settings
+    
+        err = f"Provider {self.provider} does not implement the reinforcement learning interface."
+        assert self.provider.reinforceable, err
+
+        job = self.provider.ReinforceJob(lm=self, train_kwargs=train_kwargs)
+        job.initialize()
+        return job
+
     def _run_finetune_job(self, job: TrainingJob):
         # TODO(enhance): We should listen for keyboard interrupts somewhere.
         # Requires TrainingJob.cancel() to be implemented for each provider.