diff --git a/examples/alphaevolve_interop/README.md b/examples/alphaevolve_interop/README.md index 1b95c4e7b..243174e30 100644 --- a/examples/alphaevolve_interop/README.md +++ b/examples/alphaevolve_interop/README.md @@ -16,8 +16,9 @@ check. These helpers do not replace the cloud SDK or implement all of #468. The first SDK-entrypoint slice is explicitly namespaced under `openevolve.compat.alpha_evolve`. It provides a small, synchronous lifecycle backed by OpenEvolve's public `run_evolution` API, without installing Google's -SDK or reading Google credentials. It does **not** register a top-level -`alpha_evolve` alias or change imports in other code. +SDK or reading Google credentials. Importing the facade does **not** register a +top-level `alpha_evolve` alias or change imports in other code. Scoped aliases +require the explicit opt-in described below. This self-contained example runs only a seed by default: @@ -109,15 +110,101 @@ failure insight. `run_controller_loop` only checks successful completion and accepts no cloud worker or timeout options. This is **not yet** enough to execute the original three upstream entrypoints -unchanged: SDK model types, visualization helpers, import aliases, complete -archive access and remote client/controller methods remain outside this slice. -The earlier runner below remains available for unchanged upstream evaluators. +unchanged: complete archive access, remote client/controller methods and the +full SDK model surface remain unsupported. Local connection settings are still +required, and cloud-specific configuration/controller calls need adaptation. +The runner below remains available for unchanged upstream evaluators. No managed Google Cloud run is implied by a local facade run. ```bash python -m unittest discover -s tests -p 'test_alphaevolve_local_experiment.py' -v ``` +## Load evaluator imports without Google's SDK + +Five dependency-free evaluation envelopes now cover the local examples' +`alpha_evolve.models` imports: `AlphaEvolveEvaluationScore`, +`AlphaEvolveEvaluationScores`, `AlphaEvolveEvaluationInsight`, +`AlphaEvolveEvaluationInsights` and `AlphaEvolveProgramEvaluation`. They live in +`openevolve.compat.alpha_evolve.models`. These are dataclasses with attribute +access and a no-argument `model_dump()` returning independent nested dictionaries, +including optional `None` fields. They do **not** implement Pydantic validation, +coercion, schemas, `model_validate` or serialization options. The existing +evaluator adapter continues to validate metrics and map failures. + +Load an unchanged, trusted upstream evaluator with the local types explicitly: + +```python +from pathlib import Path +from openevolve.compat.alpha_evolve import load_evaluator +from openevolve.compat.alphaevolve import adapt_evaluator + +example = Path("../alphaevolve-reference/examples/circle_packing") +evaluate = load_evaluator(example / "src/evaluate.py", "circle_packing_evaluation") +adapted = adapt_evaluator(evaluate, objective_metric="sum_of_radii") +print(adapted(example / "src/program.py").metrics) +# Or pass evaluate to the local AlphaEvolveExperiment from the previous section. +``` + +The helper reads and executes the selected `.py` file, including its import-time +code, under temporary local aliases. It returns the raw candidate-based callable +and removes both the aliases and temporary file-module registration. Registration +during import supports decorators such as `@dataclass`; removal afterwards lets +cloudpickle transport the evaluator by value to fresh OE workers, whose local +envelope classes remain importable under the `openevolve` namespace. No SDK alias +needs to remain active during evolution. The helper uses the current source +bytes rather than timestamp/size-based cached bytecode. + +Evaluator dependencies such as NumPy, SciPy and Matplotlib must still be +installed. Relative/sibling imports are not managed. Import-time side effects +outside the temporary module entries are not undone, and candidate/evaluator +code is trusted with the current process's privileges. This adds no sandbox. + +For explicit import migration, `local_imports` exposes temporary `client`, +`experiment`, `controller`, `models` and `visualization` modules: + +```python +from openevolve.compat.alpha_evolve import local_imports + +with local_imports(): + from alpha_evolve.client import AlphaEvolveClient + from alpha_evolve.visualization import get_score +# Existing references still work; sys.modules aliases have been removed. +``` + +This context is process-global, main-thread-only and intended for a +single-threaded entrypoint, not a concurrent service. It refuses to enter if +**any** `alpha_evolve` namespace is already loaded, even a nested local context; +use a fresh process rather than shadowing a real SDK. An installed but unimported +SDK is temporarily bypassed only inside the explicit context. Unknown SDK +submodules do not fall back to the cloud package. Exit, including on exception +or interruption, removes only module entries still owned by the context; it +never overwrites replacements made by other code. It does not install packages, +change import paths or credentials, or construct clients. The aliased local +client still requires explicit OE endpoint/objective settings. For evaluator +files, prefer `load_evaluator` so worker serialization does not depend on aliases. + +`openevolve.compat.alpha_evolve.visualization.get_score(program, metric)` reads +mapping or attribute-based result envelopes without plotting or executing code. +It preserves finite scores (including zero and negatives), returning `-inf` for +absent, malformed, boolean or non-finite scores. + +The existing CLI also accepts `--local-model-types` as a scoped opt-in, including +with seed-only evaluation or imported/OE evolution configs. Without that flag, +its existing Google model imports remain unchanged: + +```bash +# Requires the example's scientific dependencies, but no Google's SDK source path. +MPLBACKEND=Agg python examples/alphaevolve_interop/run.py \ + ../alphaevolve-reference/examples/circle_packing --local-model-types + +python -m unittest discover -s tests -p 'test_alphaevolve_local_imports.py' -v +``` + +This closes the local evaluator-type/import gap, not the complete SDK shim or +the original-entrypoint acceptance criterion for #468. It neither enables cloud +execution nor changes the facade's budget, concurrency or best-only query limits. + ## Evaluate an upstream seed locally From an installed OpenEvolve checkout: diff --git a/examples/alphaevolve_interop/run.py b/examples/alphaevolve_interop/run.py index 3575a7bf2..68e848d00 100644 --- a/examples/alphaevolve_interop/run.py +++ b/examples/alphaevolve_interop/run.py @@ -16,9 +16,14 @@ } -def load_evaluator(example_dir): +def load_evaluator(example_dir, *, local_model_types=False): """Load the example without importing its cloud client or experiment entrypoint.""" function_name, objective = EXAMPLES[example_dir.name] + if local_model_types: + from openevolve.compat.alpha_evolve import load_evaluator as load_local_evaluator + + evaluator = load_local_evaluator(example_dir / "src" / "evaluate.py", function_name) + return adapt_evaluator(evaluator, objective_metric=objective) spec = importlib.util.spec_from_file_location( "_alphaevolve_example", example_dir / "src" / "evaluate.py" ) @@ -34,6 +39,11 @@ def load_evaluator(example_dir): def main(argv=None): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("example_dir", type=Path, help="Path to the upstream example directory") + parser.add_argument( + "--local-model-types", + action="store_true", + help="Explicitly load evaluator imports with local envelopes instead of Google's SDK", + ) sources = parser.add_mutually_exclusive_group() sources.add_argument( "--config", type=Path, help="OE YAML config; starts evolution (legacy mode)" @@ -115,7 +125,7 @@ def main(argv=None): for path in (program_path, example_dir / "src" / "evaluate.py"): if not path.is_file(): parser.error(f"Required example file not found: {path}") - evaluator = load_evaluator(example_dir) + evaluator = load_evaluator(example_dir, local_model_types=args.local_model_types) if not evolving: result = evaluator(program_path) print(json.dumps({"metrics": result.metrics, "artifacts": result.artifacts}, indent=2)) diff --git a/openevolve/compat/alpha_evolve/__init__.py b/openevolve/compat/alpha_evolve/__init__.py index 962c1a9f4..e09f0bff4 100644 --- a/openevolve/compat/alpha_evolve/__init__.py +++ b/openevolve/compat/alpha_evolve/__init__.py @@ -1,7 +1,14 @@ -"""Explicit local experiment facade; does not install an ``alpha_evolve`` alias.""" +"""Local experiment facade; import aliases require an explicit scoped opt-in.""" from .client import AlphaEvolveClient from .controller import run_controller_loop from .experiment import AlphaEvolveExperiment +from .imports import load_evaluator, local_imports -__all__ = ["AlphaEvolveClient", "AlphaEvolveExperiment", "run_controller_loop"] +__all__ = [ + "AlphaEvolveClient", + "AlphaEvolveExperiment", + "run_controller_loop", + "load_evaluator", + "local_imports", +] diff --git a/openevolve/compat/alpha_evolve/imports.py b/openevolve/compat/alpha_evolve/imports.py new file mode 100644 index 000000000..ce1af57ce --- /dev/null +++ b/openevolve/compat/alpha_evolve/imports.py @@ -0,0 +1,83 @@ +"""Explicit, temporary import aliases and by-value evaluator loading.""" + +import importlib +import importlib.util +import sys +import threading +import uuid +from contextlib import contextmanager +from importlib.machinery import ModuleSpec +from pathlib import Path +from types import ModuleType +from typing import Callable, Iterator, Union + + +@contextmanager +def local_imports() -> Iterator[None]: + """Temporarily route ``alpha_evolve`` imports to the local facade. + + This opt-in context is process-global and main-thread-only, not thread + isolation. Use it in a single-threaded entrypoint, not a concurrent service. + Refuse any pre-existing SDK namespace (including nested local contexts). + Never change installed packages, import paths, files, or credentials. + Only this context's own module entries are removed on exit. + """ + if threading.current_thread() is not threading.main_thread(): + raise RuntimeError("Use local_imports on the main thread in a single-threaded entrypoint") + if any(name == "alpha_evolve" or name.startswith("alpha_evolve.") for name in sys.modules): + raise RuntimeError("alpha_evolve is already loaded; use a fresh process for local imports") + + modules = { + name: importlib.import_module(f"openevolve.compat.alpha_evolve.{name}") + for name in ("client", "controller", "experiment", "models", "visualization") + } + package = ModuleType("alpha_evolve", "Explicit, temporary OpenEvolve local facade") + package.__package__ = "alpha_evolve" + package.__path__ = [] + package.__spec__ = ModuleSpec("alpha_evolve", loader=None, is_package=True) + for name, module in modules.items(): + setattr(package, name, module) + aliases = {"alpha_evolve": package} + aliases.update({f"alpha_evolve.{name}": module for name, module in modules.items()}) + sys.modules.update(aliases) + try: + yield + finally: + for name, module in aliases.items(): + if sys.modules.get(name) is module: + del sys.modules[name] + + +def load_evaluator(path: Union[str, Path], function_name: str) -> Callable: + """Import a trusted Python evaluator with local model types, then remove aliases. + + Return the raw candidate-based callable; adapt it with ``adapt_evaluator`` + or pass it to the local ``AlphaEvolveExperiment``. The file module is removed + after import so cloudpickle transports the callable by value to OE workers. + Relative/sibling imports and arbitrary import side effects are not managed. + """ + path = Path(path).resolve() + if path.suffix != ".py" or not path.is_file(): + raise ValueError("path must name an existing Python .py evaluator file") + if not isinstance(function_name, str) or not function_name.isidentifier(): + raise ValueError("function_name must be a Python identifier") + name = "_openevolve_local_ae_evaluator_" + uuid.uuid4().hex + spec = importlib.util.spec_from_file_location(name, path) + if spec is None or spec.loader is None: + raise ImportError("Cannot load the evaluator file") + module = importlib.util.module_from_spec(spec) + with local_imports(): + # Registration during import supports decorators such as @dataclass. + # Remove it afterwards to select cloudpickle's by-value transport. + sys.modules[name] = module + try: + # Read the requested source, not timestamp/size-based cached bytecode: + # callers may have just edited a same-size evaluator in this second. + exec(compile(path.read_bytes(), str(path), "exec"), module.__dict__) + evaluator = getattr(module, function_name, None) + if not callable(evaluator): + raise ValueError("function_name must select a callable in the evaluator file") + finally: + if sys.modules.get(name) is module: + del sys.modules[name] + return evaluator diff --git a/openevolve/compat/alpha_evolve/models.py b/openevolve/compat/alpha_evolve/models.py new file mode 100644 index 000000000..36f458159 --- /dev/null +++ b/openevolve/compat/alpha_evolve/models.py @@ -0,0 +1,52 @@ +"""Dependency-free evaluation envelopes used by local AlphaEvolve examples. + +These dataclasses support attribute access and a no-argument ``model_dump``. +They do not emulate Pydantic validation, coercion, schemas or serialization +options. The evaluator adapter validates scores before OpenEvolve uses them. +""" + +from dataclasses import asdict, dataclass +from typing import Any, Optional + + +class _Envelope: + def model_dump(self) -> dict[str, Any]: + """Return an independent nested dictionary, including optional None fields.""" + return asdict(self) + + +@dataclass +class AlphaEvolveEvaluationScore(_Envelope): + metric: str + score: Optional[float] = None + + +@dataclass +class AlphaEvolveEvaluationScores(_Envelope): + scores: list[AlphaEvolveEvaluationScore] + + +@dataclass +class AlphaEvolveEvaluationInsight(_Envelope): + label: str + text: str + + +@dataclass +class AlphaEvolveEvaluationInsights(_Envelope): + insights: list[AlphaEvolveEvaluationInsight] + + +@dataclass +class AlphaEvolveProgramEvaluation(_Envelope): + scores: AlphaEvolveEvaluationScores + insights: Optional[AlphaEvolveEvaluationInsights] = None + + +__all__ = [ + "AlphaEvolveEvaluationScore", + "AlphaEvolveEvaluationScores", + "AlphaEvolveEvaluationInsight", + "AlphaEvolveEvaluationInsights", + "AlphaEvolveProgramEvaluation", +] diff --git a/openevolve/compat/alpha_evolve/visualization.py b/openevolve/compat/alpha_evolve/visualization.py new file mode 100644 index 000000000..76364c971 --- /dev/null +++ b/openevolve/compat/alpha_evolve/visualization.py @@ -0,0 +1,31 @@ +"""Score lookup for local result envelopes; no plotting dependencies.""" + +import math +from typing import Any + +from openevolve.compat.alphaevolve import _field + + +def get_score(program: Any, metric_name: str) -> float: + """Read a finite named score, or -inf for absent/invalid values. + + Accepts mappings or attribute-based envelopes. Unlike a plotting helper, + this does not execute a candidate or import visualization libraries. + """ + scores = _field(_field(_field(program, "evaluation"), "scores"), "scores") + if not isinstance(scores, (list, tuple)): + return -math.inf + for entry in scores: + if _field(entry, "metric") != metric_name: + continue + value = _field(entry, "score") + if value is None: + continue + if isinstance(value, bool): + return -math.inf + try: + score = float(value) + except (TypeError, ValueError, OverflowError): + return -math.inf + return score if math.isfinite(score) else -math.inf + return -math.inf diff --git a/tests/test_alphaevolve_local_imports.py b/tests/test_alphaevolve_local_imports.py new file mode 100644 index 000000000..7614b2600 --- /dev/null +++ b/tests/test_alphaevolve_local_imports.py @@ -0,0 +1,363 @@ +"""Scoped local imports and callable transport without the Google SDK.""" + +import importlib +import json +import math +import os +import pickle +import py_compile +import subprocess +import sys +import tempfile +import threading +import unittest +from pathlib import Path +from types import ModuleType, SimpleNamespace +from unittest.mock import patch + +import cloudpickle + +from openevolve.compat.alpha_evolve import ( + AlphaEvolveClient, + AlphaEvolveExperiment, + load_evaluator, + local_imports, + run_controller_loop, +) +from openevolve.compat.alpha_evolve.models import ( + AlphaEvolveEvaluationInsight, + AlphaEvolveEvaluationInsights, + AlphaEvolveEvaluationScore, + AlphaEvolveEvaluationScores, + AlphaEvolveProgramEvaluation, +) +from openevolve.compat.alpha_evolve.visualization import get_score +from openevolve.compat.alphaevolve import adapt_evaluator + + +def sdk_namespace(): + return { + name: value + for name, value in sys.modules.items() + if name == "alpha_evolve" or name.startswith("alpha_evolve.") + } + + +EVALUATOR = """from __future__ import annotations +from dataclasses import dataclass +from alpha_evolve.models import ( + AlphaEvolveEvaluationScore, AlphaEvolveEvaluationScores, + AlphaEvolveEvaluationInsight, AlphaEvolveEvaluationInsights, + AlphaEvolveProgramEvaluation, +) + +@dataclass +class EvaluationInput: + score: float + +def evaluate(candidate): + namespace = {} + exec(candidate["content"]["files"][0]["content"], namespace) + return AlphaEvolveProgramEvaluation( + scores=AlphaEvolveEvaluationScores(scores=[ + AlphaEvolveEvaluationScore(metric="quality", score=EvaluationInput(namespace["x"]).score) + ]), + insights=AlphaEvolveEvaluationInsights(insights=[ + AlphaEvolveEvaluationInsight(label="Hint", text="中文 explanation") + ]), + ) +""" + + +class TestLocalAlphaEvolveModels(unittest.TestCase): + def test_nested_dump_preserves_scores_insights_and_is_independent(self): + model = AlphaEvolveProgramEvaluation( + scores=AlphaEvolveEvaluationScores( + scores=[AlphaEvolveEvaluationScore(metric="quality", score=-4.5)] + ), + insights=AlphaEvolveEvaluationInsights( + insights=[AlphaEvolveEvaluationInsight(label="Hint", text="中文")] + ), + ) + dumped = model.model_dump() + self.assertEqual( + dumped, + { + "scores": {"scores": [{"metric": "quality", "score": -4.5}]}, + "insights": {"insights": [{"label": "Hint", "text": "中文"}]}, + }, + ) + dumped["scores"]["scores"][0]["score"] = 999 + self.assertEqual(model.scores.scores[0].score, -4.5) + self.assertIn("中文", json.dumps(model.model_dump(), ensure_ascii=False)) + + def test_null_defaults_match_evaluator_failure_envelope(self): + evaluation = AlphaEvolveProgramEvaluation( + scores=AlphaEvolveEvaluationScores( + scores=[AlphaEvolveEvaluationScore(metric="quality")] + ) + ) + self.assertEqual( + evaluation.model_dump(), + { + "scores": {"scores": [{"metric": "quality", "score": None}]}, + "insights": None, + }, + ) + + def test_model_or_dump_are_validated_by_existing_adapter(self): + with tempfile.TemporaryDirectory() as directory: + program = Path(directory) / "program.py" + program.write_text("x = 1\n") + for value in (-3.0, None, -1e12, math.nan, math.inf): + model = AlphaEvolveProgramEvaluation( + scores=AlphaEvolveEvaluationScores( + scores=[AlphaEvolveEvaluationScore(metric="quality", score=value)] + ) + ) + for result in (model, model.model_dump()): + with self.subTest(value=value, result_type=type(result)): + adapted = adapt_evaluator( + lambda candidate: result, objective_metric="quality" + ) + converted = adapted(program) + expected = -3.0 if value == -3.0 else -1e12 + self.assertEqual(converted.metrics["combined_score"], expected) + self.assertEqual("error" in converted.metrics, expected != -3.0) + + def test_dataclasses_do_not_silently_coerce_invalid_scores(self): + with tempfile.TemporaryDirectory() as directory: + program = Path(directory) / "program.py" + program.write_text("x = 1\n") + for score in ("2", True): + model = AlphaEvolveProgramEvaluation( + scores=AlphaEvolveEvaluationScores( + scores=[AlphaEvolveEvaluationScore(metric="quality", score=score)] + ) + ) + with self.subTest(score=score), self.assertRaises(TypeError): + adapt_evaluator(lambda candidate: model, objective_metric="quality")(program) + + def test_score_lookup_accepts_dicts_and_attribute_envelopes(self): + evaluation = AlphaEvolveProgramEvaluation( + scores=AlphaEvolveEvaluationScores( + scores=[ + AlphaEvolveEvaluationScore(metric="quality", score=-2.5), + AlphaEvolveEvaluationScore(metric="zero", score=0), + ] + ) + ) + for program in ( + {"evaluation": evaluation.model_dump()}, + SimpleNamespace(evaluation=evaluation), + ): + with self.subTest(program_type=type(program)): + self.assertEqual(get_score(program, "quality"), -2.5) + self.assertEqual(get_score(program, "zero"), 0) + self.assertEqual(get_score(program, "absent"), -math.inf) + + def test_score_lookup_treats_malformed_and_nonfinite_values_as_missing(self): + for program in (None, {}, {"evaluation": None}, {"evaluation": {"scores": 123}}): + with self.subTest(program=program): + self.assertEqual(get_score(program, "quality"), -math.inf) + for value in (None, "bad", {}, True, math.nan, math.inf, -math.inf): + program = { + "evaluation": {"scores": {"scores": [{"metric": "quality", "score": value}]}} + } + with self.subTest(value=value): + self.assertEqual(get_score(program, "quality"), -math.inf) + + +class TestLocalAlphaEvolveImports(unittest.TestCase): + def setUp(self): + self.before = sdk_namespace() + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name) + self.evaluator_file = self.root / "evaluate.py" + self.evaluator_file.write_text(EVALUATOR, encoding="utf-8") + + def tearDown(self): + self.assertEqual(sdk_namespace(), self.before) + self.assertFalse( + any(name.startswith("_openevolve_local_ae_evaluator_") for name in sys.modules) + ) + + def test_all_supported_aliases_resolve_only_inside_context(self): + with patch("socket.socket.connect", side_effect=AssertionError("network")): + with local_imports(): + package = importlib.import_module("alpha_evolve") + client = importlib.import_module("alpha_evolve.client") + experiment = importlib.import_module("alpha_evolve.experiment") + controller = importlib.import_module("alpha_evolve.controller") + models = importlib.import_module("alpha_evolve.models") + visualization = importlib.import_module("alpha_evolve.visualization") + self.assertIs(client.AlphaEvolveClient, AlphaEvolveClient) + self.assertIs(experiment.AlphaEvolveExperiment, AlphaEvolveExperiment) + self.assertIs(controller.run_controller_loop, run_controller_loop) + self.assertIs(models.AlphaEvolveEvaluationScore, AlphaEvolveEvaluationScore) + self.assertIs(visualization.get_score, get_score) + self.assertIs(package.models, models) + with self.assertRaises(ModuleNotFoundError): + importlib.import_module("alpha_evolve.cloud_only") + self.assertEqual(sdk_namespace(), self.before) + + def test_exception_and_interrupt_remove_owned_aliases(self): + for exception in (RuntimeError("import failed"), KeyboardInterrupt()): + with self.subTest(exception=type(exception)), self.assertRaises(type(exception)): + with local_imports(): + raise exception + self.assertEqual(sdk_namespace(), self.before) + + def test_preexisting_root_submodule_or_none_placeholder_is_never_replaced(self): + for name, value in ( + ("alpha_evolve", ModuleType("alpha_evolve")), + ("alpha_evolve.models", ModuleType("alpha_evolve.models")), + ("alpha_evolve", None), + ): + with self.subTest(name=name, value=value), patch.dict(sys.modules, {name: value}): + with self.assertRaisesRegex(RuntimeError, "already loaded"): + with local_imports(): + self.fail("must not enter") + self.assertIs(sys.modules[name], value) + + def test_nested_context_rejection_preserves_outer_aliases(self): + with local_imports(): + outer = sdk_namespace() + with self.assertRaisesRegex(RuntimeError, "already loaded"): + with local_imports(): + self.fail("must not enter") + self.assertEqual(sdk_namespace(), outer) + + def test_context_is_repeatable_after_cleanup(self): + for _ in range(2): + with local_imports(): + self.assertIs( + importlib.import_module("alpha_evolve.models").AlphaEvolveProgramEvaluation, + AlphaEvolveProgramEvaluation, + ) + self.assertEqual(sdk_namespace(), self.before) + + def test_cleanup_does_not_remove_replacements_owned_by_other_code(self): + replacement = ModuleType("replacement") + try: + with local_imports(): + sys.modules["alpha_evolve.models"] = replacement + self.assertIs(sys.modules["alpha_evolve.models"], replacement) + self.assertNotIn("alpha_evolve", sys.modules) + finally: + if sys.modules.get("alpha_evolve.models") is replacement: + del sys.modules["alpha_evolve.models"] + + def test_worker_thread_rejected_before_import_side_effects(self): + errors = [] + + def enter(): + try: + with local_imports(): + self.fail("must not enter") + except RuntimeError as error: + errors.append(str(error)) + + worker = threading.Thread(target=enter) + worker.start() + worker.join(timeout=5) + self.assertEqual(len(errors), 1) + self.assertIn("main thread", errors[0]) + + def test_file_loader_handles_dataclass_decorators_and_removes_import_names(self): + with patch("socket.socket.connect", side_effect=AssertionError("network")): + evaluator = load_evaluator(self.evaluator_file, "evaluate") + self.assertNotIn(evaluator.__module__, sys.modules) + result = evaluator({"content": {"files": [{"content": "x = -3.0\n"}]}}) + self.assertIsInstance(result, AlphaEvolveProgramEvaluation) + self.assertEqual(result.scores.scores[0].score, -3.0) + self.assertEqual(result.insights.insights[0].text, "中文 explanation") + + def test_file_loader_cleans_up_import_failures_and_missing_callables(self): + for source, name, exception in ( + ("raise RuntimeError('failed import')", "evaluate", RuntimeError), + ("raise KeyboardInterrupt()", "evaluate", KeyboardInterrupt), + ("evaluate = 123", "evaluate", ValueError), + ("def evaluate(candidate): pass", "missing", ValueError), + ("from .sibling import evaluate", "evaluate", ImportError), + ): + self.evaluator_file.write_text(source) + with self.subTest(source=source), self.assertRaises(exception): + load_evaluator(self.evaluator_file, name) + self.assertEqual(sdk_namespace(), self.before) + + def test_file_loader_uses_current_source_even_with_same_timestamp_size_cached_bytecode(self): + self.evaluator_file.write_text("def evaluate(candidate): return 1\n") + py_compile.compile(str(self.evaluator_file), doraise=True) + before = self.evaluator_file.stat() + self.assertEqual(load_evaluator(self.evaluator_file, "evaluate")({}), 1) + self.evaluator_file.write_text("def evaluate(candidate): return 2\n") + os.utime(self.evaluator_file, ns=(before.st_atime_ns, before.st_mtime_ns)) + self.assertEqual(self.evaluator_file.stat().st_size, before.st_size) + self.assertEqual(load_evaluator(self.evaluator_file, "evaluate")({}), 2) + + def test_file_loader_rejects_bad_options_before_executing_file(self): + self.evaluator_file.write_text("raise AssertionError('must not execute')") + other = self.root / "evaluate.txt" + other.write_text(EVALUATOR) + for path, name in ( + (self.root / "missing.py", "evaluate"), + (other, "evaluate"), + (self.evaluator_file, ""), + (self.evaluator_file, "bad.name"), + (self.evaluator_file, None), + ): + with self.subTest(path=path, name=name), self.assertRaises(ValueError): + load_evaluator(path, name) + + def test_file_loader_does_not_overwrite_already_imported_cloud_sdk(self): + sdk = ModuleType("alpha_evolve") + self.evaluator_file.write_text("raise AssertionError('must not execute')") + with patch.dict(sys.modules, {"alpha_evolve": sdk}): + with self.assertRaisesRegex(RuntimeError, "already loaded"): + load_evaluator(self.evaluator_file, "evaluate") + self.assertIs(sys.modules["alpha_evolve"], sdk) + + def test_loaded_callable_survives_standard_pickle_load_in_fresh_worker_without_sdk(self): + evaluator = load_evaluator(self.evaluator_file, "evaluate") + adapted = adapt_evaluator(evaluator, objective_metric="quality") + payload = self.root / "evaluator.pkl" + payload.write_bytes(cloudpickle.dumps(adapted)) + program = self.root / "program.py" + program.write_text("x = -7.5\n") + worker = """import importlib.abc, json, pickle, socket, sys +class NoCloudSDK(importlib.abc.MetaPathFinder): + def find_spec(self, fullname, path=None, target=None): + if fullname == "alpha_evolve" or fullname.startswith("alpha_evolve."): + raise AssertionError("worker must not import cloud SDK or temporary aliases") +sys.meta_path.insert(0, NoCloudSDK()) +def no_network(*args, **kwargs): + raise AssertionError("worker must not connect") +socket.socket.connect = no_network +with open(sys.argv[1], "rb") as stream: + evaluator = pickle.load(stream) +result = evaluator(sys.argv[2]) +assert not any(n == "alpha_evolve" or n.startswith("alpha_evolve.") for n in sys.modules) +print(json.dumps({"metrics": result.metrics, "artifacts": result.artifacts})) +""" + process = subprocess.run( + [sys.executable, "-c", worker, str(payload), str(program)], + cwd=self.root, + capture_output=True, + text=True, + timeout=30, + ) + self.assertEqual(process.returncode, 0, process.stderr) + result = json.loads(process.stdout) + self.assertEqual(result["metrics"], {"quality": -7.5, "combined_score": -7.5}) + self.assertEqual( + json.loads(result["artifacts"]["alphaevolve_insights"]), + [{"label": "Hint", "text": "中文 explanation"}], + ) + # Standard pickle can also load cloudpickle's by-value callable locally. + self.assertEqual(pickle.loads(payload.read_bytes())(program).metrics["quality"], -7.5) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_alphaevolve_runner.py b/tests/test_alphaevolve_runner.py index 34da7c2da..94fa4e251 100644 --- a/tests/test_alphaevolve_runner.py +++ b/tests/test_alphaevolve_runner.py @@ -231,6 +231,64 @@ def test_missing_optional_parser_is_actionable_and_json_seed_still_works(self): self.assertIn("openevolve[alphaevolve-env]", self.assert_invalid(self.env_args)) self.assertEqual(self.invoke(self.json_args)["metrics"]["sum_of_radii"], 1) + def test_local_model_types_flag_loads_legacy_imports_without_retaining_aliases(self): + (self.example / "src/evaluate.py").write_text( + "from alpha_evolve.models import (AlphaEvolveEvaluationScore, " + "AlphaEvolveEvaluationScores, AlphaEvolveProgramEvaluation)\n" + "def circle_packing_evaluation(candidate):\n" + " return AlphaEvolveProgramEvaluation(scores=AlphaEvolveEvaluationScores(\n" + " scores=[AlphaEvolveEvaluationScore(metric='sum_of_radii', score=3)]))\n" + ) + before = { + name: value for name, value in sys.modules.items() if name.startswith("alpha_evolve") + } + result = self.invoke(["--local-model-types"]) + self.assertEqual(result["metrics"]["sum_of_radii"], 3) + self.assertEqual( + {name: value for name, value in sys.modules.items() if name.startswith("alpha_evolve")}, + before, + ) + + def test_local_model_loading_is_not_enabled_implicitly(self): + with patch( + "openevolve.compat.alpha_evolve.load_evaluator", + side_effect=AssertionError("must require explicit opt-in"), + ): + self.assertEqual(self.invoke([])["metrics"]["sum_of_radii"], 1) + + def test_local_model_types_flag_runs_real_evolution_after_alias_cleanup(self): + source = self.example / "src/evaluate.py" + source.write_text( + "from alpha_evolve.models import AlphaEvolveProgramEvaluation\n" + source.read_text() + ) + + def offline_config(*args, **kwargs): + config = config_from_alphaevolve_env(*args, **kwargs) + config.database.num_islands = 1 + for model in config.llm.models + config.llm.evaluator_models: + model.init_client = offline_model + return config + + self.env.write_text("MODEL=offline-model\nMAX_PROGRAMS_GENERATED=1\n") + with ( + patch( + "openevolve.compat.alphaevolve_env.config_from_alphaevolve_env", + side_effect=offline_config, + ), + patch("socket.socket.connect", side_effect=AssertionError("network")), + ): + result = self.invoke( + [ + *self.env_args, + "--local-model-types", + "--evolve", + "--output-dir", + str(self.root / "local-output"), + ] + ) + self.assertEqual(result["metrics"]["sum_of_radii"], 2) + self.assertFalse(any(name.startswith("alpha_evolve") for name in sys.modules)) + def test_missing_seed_or_evaluator_is_reported_before_import(self): (self.example / "src/program.py").unlink() self.assertIn("Required example file", self.assert_invalid([]))