diff --git a/README.md b/README.md index 5ba3e9e..7052dab 100644 --- a/README.md +++ b/README.md @@ -86,15 +86,20 @@ cd eoapi-workshop docker compose up ``` -This will start up 6 services: +This will start up 10 services: - pgstac: postgres database with pgstac installed, running on port 5439 -- stac-fastapi-pgstac: STAC API available on port 8081 +- stac-fastapi-pgstac: upstream STAC API available directly on port 8081 +- stac-auth-proxy: primary STAC API entry point available on port 8084 +- mock-oidc-server: local test identity provider available on port 8085 - titiler-pgstac: dynamic tiler available on port 8082 - tipg: vector feature/tile server available on port 8083 -- stac-browser: beautiful interface for browsing a STAC API available on port 8085 +- stac-browser: beautiful interface for browsing a STAC API available on port 8080 +- stac-manager: web UI for editing STAC metadata via authenticated transactions on port 8086 - Jupyter Hub: interactive compute environment where you can browse the tutorial materials interactively, available on port 8888 +The local STAC API is available at `http://localhost:8084` through stac-auth-proxy. Read operations are public; transaction writes require a bearer token from the mock OIDC server (any username such as `test-user`). See [chapter 3](./docs/03-stac_fastapi_pgstac.ipynb) for read-only STAC API exploration and [chapter 6](./docs/06-stac_transactions_auth.ipynb) for authenticated transactions. [STAC Manager](http://localhost:8086) uses the same API. + 4. Open the Jupyter Hub in your web browser at `http://localhost:8888` and go through the tutorials in the `/docs` folder! ## Deploying to AWS diff --git a/docker-compose.yml b/docker-compose.yml index 4884cc4..565ae30 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -73,6 +73,7 @@ services: - POSTGRES_PORT=5432 - DB_MIN_CONN_SIZE=1 - DB_MAX_CONN_SIZE=1 + - ENABLE_TRANSACTIONS_EXTENSIONS=1 depends_on: database: condition: service_started @@ -83,6 +84,29 @@ services: volumes: - ./dockerfiles/scripts:/tmp/scripts + mock-oidc: + image: ghcr.io/alukach/mock-oidc-server:latest + ports: + - 8085:8888 + environment: + - ISSUER=http://localhost:8085 + - PORT=8888 + - SCOPES=stac:read,stac:write + + stac-auth-proxy: + image: ghcr.io/developmentseed/stac-auth-proxy:latest + ports: + - 8084:8000 + environment: + - UPSTREAM_URL=http://stac-fastapi:8081 + - OIDC_DISCOVERY_URL=http://localhost:8085/.well-known/openid-configuration + - OIDC_DISCOVERY_INTERNAL_URL=http://mock-oidc:8888/.well-known/openid-configuration + - DEFAULT_PUBLIC=true + - WAIT_FOR_UPSTREAM=true + depends_on: + - stac-fastapi + - mock-oidc + titiler-pgstac: platform: linux/amd64 image: ghcr.io/stac-utils/titiler-pgstac:1.9.0 @@ -150,11 +174,29 @@ services: ports: - 8080:8080 environment: - SB_catalogUrl: "http://0.0.0.0:8081" + SB_catalogUrl: "http://localhost:8084" depends_on: - - stac-fastapi + - stac-auth-proxy - database + stac-manager: + # no arm64 manifest published — run the amd64 image under emulation + platform: linux/amd64 + image: ghcr.io/developmentseed/stac-manager:1.0.3 + ports: + - 8086:8080 + environment: + - PUBLIC_URL=http://localhost:8086 + - REACT_APP_STAC_API=http://localhost:8084 + - REACT_APP_STAC_BROWSER=http://localhost:8080 + - REACT_APP_OIDC_AUTHORITY=http://localhost:8085 + - REACT_APP_OIDC_CLIENT_ID=stac-manager + - APP_TITLE=STAC Manager + - APP_DESCRIPTION=Manage STAC collections and items in the eoAPI workshop stack + depends_on: + - stac-auth-proxy + - mock-oidc + jupyterhub: build: context: . @@ -169,10 +211,13 @@ services: - PGPASSWORD=password - PGDATABASE=postgis - PGPORT=5432 - - STAC_API_ENDPOINT=http://stac-fastapi:8081 + - STAC_API_ENDPOINT=http://stac-auth-proxy:8000 + - STAC_AUTH_PROXY_ENDPOINT=http://stac-auth-proxy:8000 + - MOCK_OIDC_ENDPOINT=http://mock-oidc:8888 - TITILER_PGSTAC_API_ENDPOINT=http://titiler-pgstac:8082 - TIPG_API_ENDPOINT=http://tipg:8083 - STAC_BROWSER_ENDPOINT=http://localhost:8080 + - STAC_MANAGER_ENDPOINT=http://localhost:8086 volumes: pgdata: diff --git a/docs/00-introduction.ipynb b/docs/00-introduction.ipynb index f4657c2..2669580 100644 --- a/docs/00-introduction.ipynb +++ b/docs/00-introduction.ipynb @@ -27,7 +27,7 @@ "## Workshop Infrastructure:\n", "\n", "* Project homepage: \n", - "* For the workshop we have deployed an eoAPI stack to AWS using eoapi-cdk (see [DEPLOYMENT.md](../DEPLOYMENT.md) for details) \n", + "* For the workshop we have deployed an eoAPI stack to a cluster using eoapi-k8s\n", " * eoAPI API endpoints: \n", " * titiler-pgstac: \n", " * stac-fastapi-pgstac: \n", @@ -401,22 +401,29 @@ "source": [ "### 5.1 Local Development with Docker\n", "\n", - "For testing and development, eoAPI provides a Docker compose setup:\n", + "This workshop includes a Docker Compose setup for running a full eoAPI stack and JupyterHub locally. See the [README](../README.md#local-development) for Docker installation and GitHub Container Registry authentication.\n", "\n", "```bash\n", - "git clone https://github.com/developmentseed/eoAPI.git\n", - "cd eoAPI\n", + "git clone https://github.com/developmentseed/eoapi-workshop.git\n", + "cd eoapi-workshop\n", "docker compose up\n", "```\n", "\n", - "This creates a local environment with all services running in containers on your laptop which you can interact with via any http client.\n", + "This starts the following services:\n", "\n", - "- stac-fastapi-pgstac: \n", + "- pgstac (PostgreSQL): port 5439\n", + "- **STAC API** (stac-auth-proxy): — primary entry point; reads are public, writes require authentication\n", + "- stac-fastapi-pgstac: — upstream API (debugging only)\n", + "- mock-oidc-server: \n", "- titiler-pgstac: \n", "- tipg: \n", - "- stac-browser: \n", + "- stac-browser: \n", + "- stac-manager: — web UI for authenticated STAC edits\n", + "- JupyterHub: — run these notebooks interactively\n", "\n", - "The pgstac database will be accessible via `psql` on port 5439:\n", + "Open JupyterHub and work through the notebooks in `/docs`. Chapter 3 covers read-only STAC API access; [chapter 6](./06-stac_transactions_auth.ipynb) exercises authenticated transactions.\n", + "\n", + "The pgstac database is accessible via `psql` on port 5439:\n", "```bash\n", "psql postgresql://username:password@localhost:5439/postgis\n", "```" diff --git a/docs/06-stac_transactions_auth.ipynb b/docs/06-stac_transactions_auth.ipynb new file mode 100644 index 0000000..f2e6ee3 --- /dev/null +++ b/docs/06-stac_transactions_auth.ipynb @@ -0,0 +1,350 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "metadata": {}, + "source": [ + "# 6. STAC Transactions with Authentication\n", + "\n", + "So far you have only *read* from the STAC API. The [Transactions extension](https://github.com/stac-api-extensions/transactions) adds the endpoints for *writing*: creating, updating, and deleting collections and items over HTTP.\n", + "\n", + "Writing to a catalog is a privileged operation, so this workshop puts an auth layer in front of the STAC API. In the docker-compose stack, requests go through [stac-auth-proxy](https://github.com/developmentseed/stac-auth-proxy) at `http://localhost:8084`:\n", + "\n", + "- **Reads** (`GET`) are public\n", + "- **Writes** (`POST` / `PUT` / `DELETE`) require a bearer token\n", + "\n", + "In this chapter you will get a token from the local identity provider, then walk through the full life cycle of a collection and an item: create it, read it back, update it, and delete it.\n", + "\n", + "
\n", + "Every step asserts the status code it expects, so this notebook also doubles as an integration test you can run headless.\n", + "
\n", + "\n", + "
\n", + "Note: This chapter needs the local docker-compose auth stack (it reads MOCK_OIDC_ENDPOINT). It will not run against the hosted workshop deployment.\n", + "
" + ] + }, + { + "cell_type": "markdown", + "id": "b2c3d4e5-f6a7-8901-bcde-f12345678901", + "metadata": {}, + "source": [ + "## 6.1 Get an access token\n", + "\n", + "To write to the API you need to prove who you are. The stack runs a small [mock OIDC server](https://github.com/alukach/mock-oidc-server) that stands in for a real identity provider (like Auth0, Keycloak, or Cognito). You ask it for a token and it hands one back — no password required, since it is only for local testing.\n", + "\n", + "The `stac_auth` helper module wraps that exchange so the notebook stays focused on STAC:\n", + "\n", + "- `require_local_auth_stack()` checks that the auth endpoints are configured and returns them\n", + "- `get_mock_oidc_token()` requests a token with `stac:read` and `stac:write` scopes\n", + "- `auth_headers(token)` builds the `Authorization: Bearer ` header you attach to write requests" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3d4e5f6-a7b8-9012-cdef-123456789012", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "import time\n", + "\n", + "import httpx\n", + "\n", + "from stac_auth import auth_headers, get_mock_oidc_token, require_local_auth_stack\n", + "\n", + "stac_api_endpoint, mock_oidc_endpoint = require_local_auth_stack()\n", + "\n", + "write_token = get_mock_oidc_token()\n", + "write_headers = auth_headers(write_token)\n", + "\n", + "print(f\"STAC API: {stac_api_endpoint}\")\n", + "print(f\"Mock OIDC: {mock_oidc_endpoint}\")\n", + "print(f\"Token (truncated): {write_token[:16]}...\")" + ] + }, + { + "cell_type": "markdown", + "id": "d4e5f6a7-b8c9-0123-def0-234567890123", + "metadata": {}, + "source": [ + "We will create one collection and one item during this walkthrough. Giving them a timestamped id keeps them unique so you can re-run the notebook without colliding with a previous run." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e5f6a7b8-c9d0-1234-ef01-345678901234", + "metadata": {}, + "outputs": [], + "source": [ + "run_id = int(time.time())\n", + "collection_id = f\"tx-workshop-{run_id}\"\n", + "item_id = f\"tx-item-{run_id}\"\n", + "\n", + "print(f\"Test collection: {collection_id}\")\n", + "print(f\"Test item: {item_id}\")" + ] + }, + { + "cell_type": "markdown", + "id": "f6a7b8c9-d0e1-2345-f012-456789012345", + "metadata": {}, + "source": [ + "## 6.2 Confirm the transactions extension is enabled\n", + "\n", + "The transactions endpoints only exist if the API was deployed with the extension turned on. As with any STAC capability, you can check the `/conformance` response before relying on it — look for conformance classes containing `transaction`.\n", + "\n", + "
\n", + "Warning: Never enable the transactions extension on a public API without an auth layer. Doing so lets anyone write to your catalog. That is exactly why this deployment sits behind stac-auth-proxy.\n", + "
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a7b8c9d0-e1f2-3456-0123-567890123456", + "metadata": {}, + "outputs": [], + "source": [ + "conformance = httpx.get(f\"{stac_api_endpoint}/conformance\", timeout=10).json()\n", + "\n", + "transaction_classes = [\n", + " uri for uri in conformance[\"conformsTo\"] if \"transaction\" in uri.lower()\n", + "]\n", + "\n", + "print(json.dumps(transaction_classes, indent=2))\n", + "assert transaction_classes, \"Expected STAC transaction conformance classes\"" + ] + }, + { + "cell_type": "markdown", + "id": "b8c9d0e1-f2a3-4567-1234-678901234567", + "metadata": {}, + "source": [ + "## 6.3 Create a collection\n", + "\n", + "Items always live inside a collection, so we create the collection first. This is our first write, so it is a good place to see the auth boundary in action.\n", + "\n", + "We `POST` the same collection twice:\n", + "\n", + "1. **Without a token** — the proxy rejects it with `401`/`403`\n", + "2. **With our bearer token** — the write succeeds with `201`\n", + "\n", + "Then we `GET` the collection back with no token, confirming reads stay public." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c9d0e1f2-a3b4-5678-2345-789012345678", + "metadata": {}, + "outputs": [], + "source": [ + "collection = {\n", + " \"id\": collection_id,\n", + " \"type\": \"Collection\",\n", + " \"stac_version\": \"1.0.0\",\n", + " \"description\": \"Temporary collection for the transactions + auth walkthrough.\",\n", + " \"license\": \"CC-BY-4.0\",\n", + " \"extent\": {\n", + " \"spatial\": {\"bbox\": [[-10, -10, 10, 10]]},\n", + " \"temporal\": {\"interval\": [[\"2024-01-01T00:00:00Z\", None]]},\n", + " },\n", + " \"links\": [],\n", + "}\n", + "\n", + "denied = httpx.post(f\"{stac_api_endpoint}/collections\", json=collection, timeout=10)\n", + "print(f\"POST /collections without token -> {denied.status_code} (rejected)\")\n", + "assert denied.status_code in (401, 403)\n", + "\n", + "created = httpx.post(\n", + " f\"{stac_api_endpoint}/collections\",\n", + " headers=write_headers,\n", + " json=collection,\n", + " timeout=10,\n", + ")\n", + "print(f\"POST /collections with token -> {created.status_code} (created)\")\n", + "assert created.status_code in (200, 201), created.text" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d0e1f2a3-b4c5-6789-3456-890123456789", + "metadata": {}, + "outputs": [], + "source": [ + "fetched = httpx.get(f\"{stac_api_endpoint}/collections/{collection_id}\", timeout=10)\n", + "print(f\"GET /collections/{{id}} (no token) -> {fetched.status_code} (public read)\")\n", + "assert fetched.status_code == 200\n", + "assert fetched.json()[\"id\"] == collection_id" + ] + }, + { + "cell_type": "markdown", + "id": "e1f2a3b4-c5d6-7890-4567-901234567890", + "metadata": {}, + "source": [ + "## 6.4 Add an item\n", + "\n", + "Now we add an item to the collection by posting a GeoJSON `Feature` to the collection's `/items` endpoint. A STAC item needs at least an `id`, a `geometry`, a `bbox`, and a `datetime` in its `properties`.\n", + "\n", + "Same auth rule applies: the unauthenticated `POST` is rejected, the authenticated one succeeds." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f2a3b4c5-d6e7-8901-5678-012345678901", + "metadata": {}, + "outputs": [], + "source": [ + "item = {\n", + " \"id\": item_id,\n", + " \"type\": \"Feature\",\n", + " \"stac_version\": \"1.0.0\",\n", + " \"collection\": collection_id,\n", + " \"geometry\": {\"type\": \"Point\", \"coordinates\": [0.5, 0.5]},\n", + " \"bbox\": [0.5, 0.5, 0.5, 0.5],\n", + " \"properties\": {\"datetime\": \"2024-06-01T00:00:00Z\"},\n", + " \"links\": [],\n", + " \"assets\": {},\n", + "}\n", + "\n", + "items_url = f\"{stac_api_endpoint}/collections/{collection_id}/items\"\n", + "\n", + "denied_item = httpx.post(items_url, json=item, timeout=10)\n", + "print(f\"POST item without token -> {denied_item.status_code} (rejected)\")\n", + "assert denied_item.status_code in (401, 403)\n", + "\n", + "created_item = httpx.post(items_url, headers=write_headers, json=item, timeout=10)\n", + "print(f\"POST item with token -> {created_item.status_code} (created)\")\n", + "assert created_item.status_code in (200, 201), created_item.text" + ] + }, + { + "cell_type": "markdown", + "id": "a3b4c5d6-e7f8-9012-6789-123456789012", + "metadata": {}, + "source": [ + "## 6.5 Update an item\n", + "\n", + "`PUT` **replaces** an item in full rather than patching a few fields. The safe pattern is therefore *get-then-modify*: fetch the current item, change what you need, and send the whole object back. This avoids accidentally dropping fields the server already stored.\n", + "\n", + "Here we add a `description` property and `PUT` the modified item back." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b4c5d6e7-f8a9-0123-7890-234567890123", + "metadata": {}, + "outputs": [], + "source": [ + "item_url = f\"{stac_api_endpoint}/collections/{collection_id}/items/{item_id}\"\n", + "\n", + "stored_item = httpx.get(item_url, timeout=10).json()\n", + "stored_item[\"properties\"][\"description\"] = \"Updated by authenticated PUT\"\n", + "\n", + "denied_put = httpx.put(item_url, json=stored_item, timeout=10)\n", + "print(f\"PUT item without token -> {denied_put.status_code} (rejected)\")\n", + "assert denied_put.status_code in (401, 403)\n", + "\n", + "updated_item = httpx.put(item_url, headers=write_headers, json=stored_item, timeout=10)\n", + "print(f\"PUT item with token -> {updated_item.status_code} (updated)\")\n", + "assert updated_item.status_code == 200, updated_item.text\n", + "assert (\n", + " updated_item.json()[\"properties\"][\"description\"] == \"Updated by authenticated PUT\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "c5d6e7f8-a9b0-1234-8901-345678901234", + "metadata": {}, + "source": [ + "## 6.6 Delete an item\n", + "\n", + "`DELETE` removes the item. Afterwards a `GET` for that item returns `404`, confirming it is gone." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d6e7f8a9-b0c1-2345-9012-456789012345", + "metadata": {}, + "outputs": [], + "source": [ + "denied_delete = httpx.delete(item_url, timeout=10)\n", + "print(f\"DELETE item without token -> {denied_delete.status_code} (rejected)\")\n", + "assert denied_delete.status_code in (401, 403)\n", + "\n", + "deleted_item = httpx.delete(item_url, headers=write_headers, timeout=10)\n", + "print(f\"DELETE item with token -> {deleted_item.status_code} (deleted)\")\n", + "assert deleted_item.status_code in (200, 204)\n", + "\n", + "missing_item = httpx.get(item_url, timeout=10)\n", + "print(f\"GET deleted item -> {missing_item.status_code} (not found)\")\n", + "assert missing_item.status_code == 404" + ] + }, + { + "cell_type": "markdown", + "id": "e7f8a9b0-c1d2-3456-0123-567890123456", + "metadata": {}, + "source": [ + "## 6.7 Clean up\n", + "\n", + "Finally, delete the temporary collection so we leave the catalog as we found it. Deleting a collection is a write too, so it also needs the token.\n", + "\n", + "You have now exercised the full transaction life cycle — create, read, update, delete — and seen the auth proxy enforce the read/write boundary at every step." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f8a9b0c1-d2e3-4567-1234-678901234567", + "metadata": {}, + "outputs": [], + "source": [ + "collection_url = f\"{stac_api_endpoint}/collections/{collection_id}\"\n", + "\n", + "denied_collection_delete = httpx.delete(collection_url, timeout=10)\n", + "print(\n", + " f\"DELETE collection without token -> {denied_collection_delete.status_code} (rejected)\"\n", + ")\n", + "assert denied_collection_delete.status_code in (401, 403)\n", + "\n", + "deleted_collection = httpx.delete(collection_url, headers=write_headers, timeout=10)\n", + "print(f\"DELETE collection with token -> {deleted_collection.status_code} (deleted)\")\n", + "assert deleted_collection.status_code in (200, 204)\n", + "\n", + "print(\"\\nAll transaction + auth checks passed.\")" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/stac_auth.py b/docs/stac_auth.py new file mode 100644 index 0000000..27be225 --- /dev/null +++ b/docs/stac_auth.py @@ -0,0 +1,71 @@ +""" +Helpers for authenticated STAC transaction requests in the local docker-compose stack. + +Requires MOCK_OIDC_ENDPOINT and STAC_API_ENDPOINT (stac-auth-proxy). +""" + +from __future__ import annotations + +import html +import json +import os +import re + +import httpx + +_TOKEN_PATTERN = re.compile(r']*id="token"[^>]*>(.*?)', re.S) + + +def stac_endpoint() -> str: + return os.getenv("STAC_API_ENDPOINT") or os.getenv("STAC_AUTH_PROXY_ENDPOINT", "") + + +def mock_oidc_endpoint() -> str | None: + return os.getenv("MOCK_OIDC_ENDPOINT") + + +def require_local_auth_stack() -> tuple[str, str]: + """Return (stac_endpoint, mock_oidc_endpoint) or raise.""" + stac = stac_endpoint() + oidc = mock_oidc_endpoint() + if not stac or not oidc: + raise RuntimeError( + "This notebook requires the docker-compose auth stack. " + "Set STAC_API_ENDPOINT and MOCK_OIDC_ENDPOINT " + "(run `docker compose up`)." + ) + return stac, oidc + + +def get_mock_oidc_token( + username: str = "test-user", + scopes: str = "openid profile stac:read stac:write", + *, + oidc_endpoint: str | None = None, + timeout: float = 10.0, +) -> str: + """Request a bearer token from the mock OIDC server.""" + oidc_endpoint = oidc_endpoint or mock_oidc_endpoint() + if not oidc_endpoint: + raise RuntimeError("MOCK_OIDC_ENDPOINT is not configured") + + response = httpx.post( + f"{oidc_endpoint.rstrip('/')}/", + data={ + "username": username, + "scopes": scopes, + "claims": json.dumps({"email": f"{username}@example.com"}), + }, + timeout=timeout, + ) + response.raise_for_status() + + match = _TOKEN_PATTERN.search(response.text) + if not match: + raise RuntimeError("Mock OIDC response did not include a token") + + return html.unescape(match.group(1)).strip() + + +def auth_headers(token: str) -> dict[str, str]: + return {"Authorization": f"Bearer {token}"}