From e5078dea7b937abc724d8afcd9123295f573a12e Mon Sep 17 00:00:00 2001 From: PGijsbers Date: Wed, 24 Dec 2025 14:38:28 +0100 Subject: [PATCH 1/5] write updated instructions and favor running containerized --- .python-version | 1 - docs/installation.md | 152 +++++-------------------------------------- 2 files changed, 15 insertions(+), 138 deletions(-) delete mode 100644 .python-version diff --git a/.python-version b/.python-version deleted file mode 100644 index e4fba21..0000000 --- a/.python-version +++ /dev/null @@ -1 +0,0 @@ -3.12 diff --git a/docs/installation.md b/docs/installation.md index b239601..679a3f1 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,144 +1,22 @@ # Installation -*Current instructions tested on Mac, but likely work on most Unix systems.* +The primary way to run this service is through a docker container. +The REST API needs to be able to connect to a MySQL database with the OpenML "openml" and "openml_expdb" databases. +The `docker-compose.yaml` file of this project defines these together out of the box. +This is useful for development purposes, but the database does not persist between restarts in the current configuration. +By default, the current code is also mounted into the Python REST API container (again, for development purposes). -The OpenML server will be developed and maintained for the latest minor release of -Python (Python 3.12 as of writing). -You can install the dependencies locally or work with docker containers. +For development, it should suffice to run the services from a fresh clone by running `docker compose --profile "python" up -d`. +The REST API will be exposed on port 8001 on the host machine. To visit the Swagger Docs, visit http://localhost:8001/docs. -??? tip "Use `pyenv` to manage Python installations" +Once the containers are started, you can run tests with `docker exec -it openml-python-rest-api python -m pytest -m "not php_api" tests`. +For migration testing, which compares output of the Python based REST API with the old PHP based one, also start the PHP server (`docker compose --profile "php" --profile "python" up -d`) and include tests with the `php_api` marker/fixture: `docker exec -it openml-python-rest-api python -m pytest tests`. - We recommend using [`pyenv`](https://github.com/pyenv/pyenv) if you are working with - multiple local Python versions. After following the installation instructions for - `pyenv` check that you can execute it: +!!! note - ```text - > pyenv local - 3.12 - ``` + The PHP REST API needs Elastic Search. In some cases, it also needs the ES indices to be built. + The current set up does not automatically build ES indices, because that takes a long time. + When we start testing more upload functionality, for which the PHP API needs built indices, we'll work on an ES image with prebuilt indices. - If `pyenv` can't be found, please make sure to update the terminal environment - (either by `reset`ing it, or by closing and opening the terminal). If you get the message - `pyenv: no local version configured for this directory` first clone the repository - as described below and try again from the root of the cloned repository. - - You can then install the Python version this project uses with: - `cat .python-version | pyenv install` - - -## Local Installation - -These instructions assume [Python 3.12](https://www.python.org/downloads/) -and [git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) are already installed. - -!!! info "You may need to install Python3 and MySQL development headers." - - It may be necessary to first install additional headers before proceeding with a - local installation of the `mysqlclient` dependency. They are documented under - ["Installation"](https://github.com/PyMySQL/mysqlclient#linux) of the `mysqlclient` - documentation. - -=== "For Users" - - If you don't plan to make code changes, you can install directly from Github. - We recommend to install the OpenML server and its dependencies into a new virtual - environment. - ```bash title="Installing the project into a new virtual environment" - python -m venv venv - source venv/bin/activate - - python -m pip install git+https://github.com/openml/server-api.git - ``` - If you do plan to make code changes, we recommend you follow the instructions - under the "For Contributors" tab, even if you do not plan to contribute your - changes back into the project. - - -=== "For Contributors" - - If you plan to make changes to this project, it will be useful to install - the project from a cloned fork. To fork the project, go to our - [project page](https://github.com/openml/server-api) and click "fork". - This makes a copy of the repository under your own Github account. - You can then clone your own fork (replace `USER_NAME` with your Github username): - - ```bash title="Cloning your fork" - git clone https://github.com/USER_NAME/server-api.git - cd server-api - ``` - - Then we can install the project into a new virtual environment in edit mode: - - ```bash title="Installing the project into a new virtual environment" - python -m venv venv - source venv/bin/activate - - python -m pip install -e ".[dev,docs]" - ``` - Note that this also installs optional dependencies for development and documentation - tools. We require this for contributors, but we also highly recommend it anyone - that plans to make code changes. - -## Setting up a Database Server -Depending on your use of the server, there are multiple ways to set up your own -OpenML database. To simply connect to an existing database, see -[configuring the REST API Server](#configuring-the-rest-api-server) below. - - -### Setting up a new database -This sets up an entirely empty database with the expected OpenML tables in place. -This is intended for new deployments of OpenML, for example to host a private OpenML -server. - -!!! Failure "" - - Instructions are incomplete. See [issue#78](https://github.com/openml/server-api/issues/78). - -### Setting up a test database - -We provide a prebuilt docker image that already contains test data. - -=== "Docker Compose" - To start the database through `docker compose`, run: - - ```bash - docker compose up database - ``` - - which starts a database. - -=== "Docker Run" - - To start a test database as stand-alone container, run: - - ```bash - docker run --rm -e MYSQL_ROOT_PASSWORD=ok -p 3306:3306 -d --name openml-test-database openml/test-database:latest - ``` - - You may opt to add the container to a network instead, to make it reachable - from other docker containers: - - ```bash - docker network create openml - docker run --rm -e MYSQL_ROOT_PASSWORD=ok -p 3306:3306 -d --name openml-test-database --network openml openml/test-database:latest - ``` - -The container may take a minute to initialise, but afterwards you can connect to it. -Either from a local `mysql` client at `127.0.0.1:3306` or from a docker container -on the same network. For example: - -```bash -docker run --network NETWORK --rm -it mysql mysql -hopenml-test-database -uroot -pok -``` -where `NETWORK` is `openml` when using `docker run` when following the example, -and `NETWORK` is `server-api_default` if you used `docker compose` (specifically, -it is `DIRECTORY_NAME` + `_default`, so if you renamed the `server-api` directory to -something else, the network name reflects that). - -## Configuring the REST API Server - -The REST API is configured through a [TOML](https://toml.io) file. - -!!! Failure "" - - Instructions are incomplete. Please have patience while we are adding more documentation. +Information for a production deployment will follow, in a nutshell you need to configure the REST API to connect to a persistent database, +which can be the one defined in `docker-compose.yaml` if has an appropriately mounted volume. From cd3aac6602f3e2679e8b69559a49dbe2d06dd77c Mon Sep 17 00:00:00 2001 From: PGijsbers Date: Wed, 24 Dec 2025 14:42:16 +0100 Subject: [PATCH 2/5] Add caveat to the migration guide about sync issues --- docs/migration.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/migration.md b/docs/migration.md index 78f8761..8a69b9f 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -13,6 +13,9 @@ There may be undocumented changes, especially in edge cases which may not have o As the PHP API was underspecified, the re-implementation is based on a mix of reading old code and probing the API. If there is a behavioral change which was not documented but affects you, please [open a bug report](https://github.com/openml/server-api/issues/new?assignees=&labels=bug%2C+triage&projects=&template=bug-report.md&title=). +It is possible this migration guide is out of sync for endpoints not yet deployed to production (currently that includes them all). +Before an endpoint is deployed to production we will ensure that the documentation is up-to-date to the best of our knowledge. + ## All Endpoints The following changes affect all endpoints. From ed597a244b482ee8587c088004ee37a6074fe245 Mon Sep 17 00:00:00 2001 From: Pieter Gijsbers Date: Thu, 25 Dec 2025 10:59:13 +0100 Subject: [PATCH 3/5] Update docs/installation.md Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --- docs/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/installation.md b/docs/installation.md index 679a3f1..6364da3 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,6 +1,6 @@ # Installation -The primary way to run this service is through a docker container. +The primary way to run this service is through a Docker container. The REST API needs to be able to connect to a MySQL database with the OpenML "openml" and "openml_expdb" databases. The `docker-compose.yaml` file of this project defines these together out of the box. This is useful for development purposes, but the database does not persist between restarts in the current configuration. From 204aafc130411314a38b3a01bceaf7b05fb07f4f Mon Sep 17 00:00:00 2001 From: Pieter Gijsbers Date: Thu, 25 Dec 2025 10:59:57 +0100 Subject: [PATCH 4/5] Update docs/installation.md Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --- docs/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/installation.md b/docs/installation.md index 6364da3..0a590d2 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -10,7 +10,7 @@ For development, it should suffice to run the services from a fresh clone by run The REST API will be exposed on port 8001 on the host machine. To visit the Swagger Docs, visit http://localhost:8001/docs. Once the containers are started, you can run tests with `docker exec -it openml-python-rest-api python -m pytest -m "not php_api" tests`. -For migration testing, which compares output of the Python based REST API with the old PHP based one, also start the PHP server (`docker compose --profile "php" --profile "python" up -d`) and include tests with the `php_api` marker/fixture: `docker exec -it openml-python-rest-api python -m pytest tests`. +For migration testing, which compares output of the Python-based REST API with the old PHP-based one, also start the PHP server (`docker compose --profile "php" --profile "python" up -d`) and include tests with the `php_api` marker/fixture: `docker exec -it openml-python-rest-api python -m pytest tests`. !!! note From 6c7ffc74a73f932aecaf789c0eed9c4584830d14 Mon Sep 17 00:00:00 2001 From: Pieter Gijsbers Date: Thu, 25 Dec 2025 11:00:09 +0100 Subject: [PATCH 5/5] Update docs/installation.md Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --- docs/installation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/installation.md b/docs/installation.md index 0a590d2..c506537 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -14,7 +14,7 @@ For migration testing, which compares output of the Python-based REST API with t !!! note - The PHP REST API needs Elastic Search. In some cases, it also needs the ES indices to be built. + The PHP REST API needs Elasticsearch. In some cases, it also needs the ES indices to be built. The current set up does not automatically build ES indices, because that takes a long time. When we start testing more upload functionality, for which the PHP API needs built indices, we'll work on an ES image with prebuilt indices.