From 07a2e17c06e5a2e3eb9a9fa91e2916aa4a4391a5 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 30 Apr 2026 12:12:35 +0100 Subject: [PATCH 01/14] update cli intro --- docs/build-for-developers/cli-intro.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/build-for-developers/cli-intro.md b/docs/build-for-developers/cli-intro.md index 944c7712318a..bf58e97c5c47 100644 --- a/docs/build-for-developers/cli-intro.md +++ b/docs/build-for-developers/cli-intro.md @@ -11,10 +11,10 @@ workflows directly from the command line. It’s simple to install, works on macOS, Windows, and Linux, and offers a range of functionality to enhance your developer experience with OpenFn. You can use the OpenFn CLI to: -- Securely run OpenFn steps and workflows +- Sync workflows between OpenFn and a local filesystem or GitHub +- Securely run OpenFn workflows - Troubleshoot and debug OpenFn steps -- Access adaptor documentation -- Deploy workflows to OpenFn +- Read and write Collections data --- @@ -25,7 +25,7 @@ Before you begin with the @openfn/cli, make sure to setup some key tooling: 1. **Code Editor:** Ensure you have a code editor installed on your machine. You can use popular editors like [VS Code](https://code.visualstudio.com/) or [Sublime](https://www.sublimetext.com/). -2. **Node.js:** Install Node.js (version 18 or later). For Linux, Windows, or +2. **Node.js:** Install Node.js (version 24 or later). For Linux, Windows, or macOS, use a version manager like [nvm](https://github.com/nvm-sh/nvm) or [asdf](https://asdf-vm.com/guide/getting-started.html). Or [install Node.js directly](https://kinsta.com/blog/how-to-install-node-js/) @@ -57,7 +57,7 @@ The word `openfn` will invoke the CLI. The word `test` will invoke the test command.
-Expand to see the expected output. +Expand to see the expected output ``` [CLI] ♦ Versions: From 34b321a7cfdc8a4c03303ce4ceaa5f9e9827a633 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 30 Apr 2026 16:56:34 +0100 Subject: [PATCH 02/14] start docs --- docs/build-for-developers/cli-sync.md | 197 ++++++++++++++++++++++++++ sidebars-main.js | 3 +- 2 files changed, 199 insertions(+), 1 deletion(-) create mode 100644 docs/build-for-developers/cli-sync.md diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md new file mode 100644 index 000000000000..5469e1cc5e46 --- /dev/null +++ b/docs/build-for-developers/cli-sync.md @@ -0,0 +1,197 @@ +--- +title: OpenFn Sync +sidebar_label: OpenFn Sync +slug: /sync +--- + +OpefnFn projects are fully portable, which means they can be moved to different +places. + +You can create a project in the app, download it to your local machine to +develop offline, re-upload it to the app, or even deploy it to a different +OpenFn server. + +We call this OpenFn Sync, and it's one of the most powerful features affording +by OpenFn Projects. + +## What is a Project? + +A Project basically set of Workflows (and maybe some configuration) which +solves, automates or integrates some business function. It usually lives in the +app, but it can just be some files on a file system. + +Inside the OpenFn app, a Project is a top level entity backed by a bunch of +database tables. + +The app Project includes a bunch of extra stuff: like configuring Channels, +tracking the run history, saving dataclips, and providing support from the AI +assistant. + +The Project can also exist on the file system, in which case it's a bunch of +files which can be read and executed by the CLI. This local representation of a +project is pretty bare-bones: you'll only find workflows and code here. + +Multiple related projects can exist on the file system at any one time. Each +will ive in a single project file. You can "check out" or "expand" one project +at a time to a local folder, which will create one file for each workflow and +one file for each step. + +Sometimes, We call this superset of all known, related, distributed projects a +Workspace. + +Not all artifacts of a Project are included in a sync. Generally we sync the +project's workflows and some of its options. But we do not sync any associated +data, credential values, usage history or AI sessions. + +## Authorization + +Before you use the CLI to fetch anything from the app, you'll need to provide +some authorization. + +The best way to do this is to set an environment variable called +`OPENFN_API_KEY`. Set it to the value of your +[Personal Access Token](https://docs.openfn.org/documentation/api-tokens#about-api-tokens). + +::: info Personal Access Tokens + +See +[Create and Manage API Tokens](https://docs.openfn.org/documentation/api-tokens) +for help setting up a token. + +::: + +If you connect to multiple OpenFn projects or apps, you can create a `.env` file +and set any environment variables you need there. The CLI will load this file +and report which keys its using. Values in your `.env` file will be preferred to +those defined in your system. + +You can also + +::: info + +This guide assumes you want to sync with our hosted SaaS app at app.openfn.org + +You can sync with a different OpenFn instance by setting the env var +`OPENFN_ENDPOINT` or by passing the `--endpoint` argument to most commands. + +::: + +## Downloading a Project + +To pull a project down from the app to your local machine, run: + +```bash +openfn project pull +``` + +After you've pulled a project for the first time, you don't need to specify the +UUID again (in any commands). + +The `project pull` command does three things: + +- If you do not have an `openfn.yaml` file, it'll create one +- It will _fetch_ your project from the app and save it into a single file at + `.projects/main@app.openfn.org.yaml` +- It will _checkout_ that project onto your file system, expanding workflows and + steps to their own files. + +This will create a file in your working directory called +`.projects/main@app.openfn.org.yaml`. + +## Understanding a Project Structure + +Pulling a project from the app will create a file structure that looks like +this: + +``` +├── openfn.yaml +├── .projects +│ ├── main@app.staging.openfn.org.yaml +└── workflows + ├── my-workflow + │ ├── my-workflow.yaml + │ ├── job.js +``` + +There are three key files to understand here: + +### project.yaml + +The project file saves a copy of the whole state of a project as saved in the +app. If you look inside you'll see the workflows represented as0 plain text. + +A project file is named like `@.yaml`. The alias is a local name +used to refer to a particular version of the project. + +The project file should not be edited locally as changes will be dropped on the +next fetch. + +You can fetch as many projects as you like, and each will be saved to its own +project.yaml file. + +The `.projects` folder can and should be committed to source control. + +### workflows + +Having your whole project inside a single file isn't actually a great way to +read or edit workflows. So the CLI can "checkout" or "expand" a project file +onto the file system. + +Checking out is the process of writing each workflow to a workflow.yaml file and +each step to a step.js file. This all lives in the `workflows` directory. + +Here you can edit files as much as you like, and changes will be tracked when +you push/deploy back to the app. + +You can only check out on project a time. This is actually great for working +with git, because you can checkout two projects on different branches and +compare/merge them against each other directly. + +### openfn.yaml + +This is a top-level configuration file + +We call this file system a Workspace (not a project, because technically it +holds multiple projects) + +At any time, one project can be "checked out" or "expanded". That means the the +contents of the `workflows` folder represents that specific project. You cannot +checkout two versions of the same workflow at once. + +- `openfn.yaml` contains settings and metadata about the checked out project +- `.projects` contains a yaml file for each project. This is a local copy of a + remote project. It should always reflect an app's view of a project. You + should generally not edit these files. +- `workflows` contains all the workflows for the checked out project. These are + compatible with the CLI. + +## Aliases + +Rather than identify a local project with a long-winded UUID or id string, you +can use an _alias_. + +Each local project is saved to a file like `main@app.openfn.org.yaml`, where the +`main` bit is the local alias of a project. + +You can pull a project and set the alias at the same time by running + +```bash +openfn project pull --alias dev +``` + +This will save the project to `dev@app.openfn.org.yaml` + +To change the alias, you can simply rename the file. Whatever goes before the +`@` will be treated as the alias. + +## Running workflows in projects + +## Deploying a Project + +## Sandboxes + +## GitHub + +## Cheatsheet + +An OpenFn project can be represented as a single yaml file diff --git a/sidebars-main.js b/sidebars-main.js index cc0bc31491ca..6cb20f464cbd 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -124,9 +124,10 @@ module.exports = { items: [ 'build-for-developers/cli-intro', 'build-for-developers/cli-usage', + 'build-for-developers/cli-sync', + 'build-for-developers/cli-collections', 'build-for-developers/cli-walkthrough', 'build-for-developers/cli-challenges', - 'build-for-developers/cli-collections', ], }, { From ceefdc4854c93806440d58708047fb7b7ff5fcd1 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 30 Apr 2026 17:21:57 +0100 Subject: [PATCH 03/14] more docs --- docs/build-for-developers/cli-sync.md | 92 ++++++++++++++++++++++++++- 1 file changed, 90 insertions(+), 2 deletions(-) diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index 5469e1cc5e46..dcafce4e74c2 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -65,7 +65,7 @@ and set any environment variables you need there. The CLI will load this file and report which keys its using. Values in your `.env` file will be preferred to those defined in your system. -You can also +You can also pass `--api-key` directly as a flag to most commands. ::: info @@ -184,14 +184,102 @@ This will save the project to `dev@app.openfn.org.yaml` To change the alias, you can simply rename the file. Whatever goes before the `@` will be treated as the alias. +## Checking Out + +You can checkout a project any time with: + +```bash +openfn project checkout +``` + +This will update your local workflows folder with the target project. + +If a checkout will cause changes to be lost (ie, you've changed a step.js file +but haven't deployed it), you will be warned. Add `--force` to ignore the +change, or run `openfn project clean` to wipe and reset the `workflows` folder. + +Checking out will only modify files managed by the CLI - basically workflow and +step files. If you have other files in the file system (like state files or test +files) they will be untouched. + ## Running workflows in projects +You can execute any workflow in the checked out project by name: + +```bash +openfn my-workflow +``` + +The CLI will find the workflow in your `workflows` folder and run it. You can +pass state via `-s` and set log levels as per usual. + +When running a workflow by name like this, you get two benefits: + +- **Credentials** are loaded automatically from the credential map in + `openfn.yaml`, so you don't need to pass `--credential-map` +- **Collections** use the server configured in `openfn.yaml`, so you don't need + to pass `--collections-endpoint` or anything. + ## Deploying a Project +To push your local changes back to the app, run: + +```bash +openfn project deploy +``` + +This will take your currently checked out project and push it to the app. It +will also report what has changed in the local project. + +Before uploading, the CLI fetches the latest version of the project from the app +and checks for **divergence** — meaning it checks whether any of your +locally-changed workflows have also been edited in the app since you last +pulled. If they have, the deploy will fail with an error to prevent you from +accidentally overwriting someone else's work. + +If you want to push anyway, pass `--force`: + +```bash +openfn project deploy --force +``` + +To preview what would change without actually uploading anything, use +`--dry-run`. This will log the final update payload that would be sent to the +app (as a JSON structure). + +You can deploy the checked out project as a new project on the target app by +adding the `--new` command. This is only available if you have superuser +privileges on the target instance. + +You can also deploy the checked out project to another app project by passing +its alias, id or uuid: + +``` +openfn project deploy main +``` + +If you currently have a development sandbox checked out, this would merge it +straight into the main app project. + +Note that you have to have fetched the target project locally before you can +deploy it. + ## Sandboxes ## GitHub ## Cheatsheet -An OpenFn project can be represented as a single yaml file +| Command | Description | +| ---------------------------------------- | ------------------------------------------------------------------------------ | ----- | --------------------------------------- | +| `openfn project pull ` | Pull a project from the app for the first time | +| `openfn project pull` | Re-pull the current project | +| `openfn project pull --alias dev` | Pull and set a local alias | +| `openfn project fetch ` | Switch to a different local project | +| `openfn project deploy` | Deploy checked-out project to the app | +| `openfn project deploy --dry-run` | Try a deploy but skip the upload step | +| `openfn project deploy --force` | Force the checked out project to be uploaded, ignoring any divergence warnings | +| `openfn ` | Run a workflow in the checked-out project | +| `openfn project clean` | Delete the `workflows` folder and all contents, then check out the project | From e5358103a5c87788c26bb020a1adc607eb0ab0c2 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 17:40:47 +0100 Subject: [PATCH 04/14] revision --- docs/build-for-developers/cli-sync.md | 130 ++++++++++++++++++-------- 1 file changed, 89 insertions(+), 41 deletions(-) diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index dcafce4e74c2..87c62e03919d 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -1,6 +1,6 @@ --- title: OpenFn Sync -sidebar_label: OpenFn Sync +sidebar_label: Sync slug: /sync --- @@ -16,28 +16,40 @@ by OpenFn Projects. ## What is a Project? -A Project basically set of Workflows (and maybe some configuration) which -solves, automates or integrates some business function. It usually lives in the -app, but it can just be some files on a file system. +A Project is a set of Workflows which solves, automates or integrates some +business function. -Inside the OpenFn app, a Project is a top level entity backed by a bunch of -database tables. +A Project lives in the OpenFn app (either on the cloud Saas instance or on a +privately deployed instance ), but it also can exist as files on a file system. + +Attached to every Project is some metadata (like a name and a description) and +some configuration like Credentials and Collections. + +Inside the OpenFn app, a Project is a billable top level entity, where all the +workflows and configuration are saved to database tables. The app Project includes a bunch of extra stuff: like configuring Channels, -tracking the run history, saving dataclips, and providing support from the AI +tracking the run history, saving dataclips, and chat sessions with the AI assistant. -The Project can also exist on the file system, in which case it's a bunch of +The Project can also exist on a local file system, in which case it's a bunch of files which can be read and executed by the CLI. This local representation of a project is pretty bare-bones: you'll only find workflows and code here. Multiple related projects can exist on the file system at any one time. Each -will ive in a single project file. You can "check out" or "expand" one project +will live in a single project file. You can "check out" or "expand" one project at a time to a local folder, which will create one file for each workflow and one file for each step. -Sometimes, We call this superset of all known, related, distributed projects a -Workspace. +It's typical that a single conceptual project - that is, the code and +configuration which drives a business function - exists in several places at +once. It might have several representations in the app through sandboxes; be +backed up to GitHub; run locally on a developer's machine; and be distributed to +several remote instances to run in production. + +Sometimes, we call this superset of all known, related, distributed Projects a +Workspace. The Sync problem is the question of how code and configuration is +copied/deployed/replicated across Project instances. Not all artifacts of a Project are included in a sync. Generally we sync the project's workflows and some of its options. But we do not sync any associated @@ -52,7 +64,7 @@ The best way to do this is to set an environment variable called `OPENFN_API_KEY`. Set it to the value of your [Personal Access Token](https://docs.openfn.org/documentation/api-tokens#about-api-tokens). -::: info Personal Access Tokens +:::info Personal Access Tokens See [Create and Manage API Tokens](https://docs.openfn.org/documentation/api-tokens) @@ -67,9 +79,10 @@ those defined in your system. You can also pass `--api-key` directly as a flag to most commands. -::: info +:::info -This guide assumes you want to sync with our hosted SaaS app at app.openfn.org +This guide assumes you want to sync with our hosted SaaS app at +[app.openfn.org](https://app.openfn.org) You can sync with a different OpenFn instance by setting the env var `OPENFN_ENDPOINT` or by passing the `--endpoint` argument to most commands. @@ -84,19 +97,34 @@ To pull a project down from the app to your local machine, run: openfn project pull ``` +This will create a file in your working directory called +`.projects/main@app.openfn.org.yaml`. + +:::info + +Every Project in the app has a unique identifier, called a UUID, which is used +to reference it. It is a 32 digit number of the form +`a6cc5bdd-b04f-4413-b4b8-132a5115acac` + +You can copy a Project's UUID from the URL by browsing to it in the app. It's +the long string after `projects`. + +For example, the UUID is the bold section in: +{'https://openfn.org/projects/'}{'abc087dd-3963-4260-8d09-ced2e1ff2bb0'}{'/w'} + +::: + After you've pulled a project for the first time, you don't need to specify the -UUID again (in any commands). +UUID again. You can use the alias, the id, or leave the identifier blank to use +the currently checked out project. The `project pull` command does three things: - If you do not have an `openfn.yaml` file, it'll create one -- It will _fetch_ your project from the app and save it into a single file at - `.projects/main@app.openfn.org.yaml` -- It will _checkout_ that project onto your file system, expanding workflows and - steps to their own files. - -This will create a file in your working directory called -`.projects/main@app.openfn.org.yaml`. +- It will _fetch_ (download) your project from the app and save it into a single + file at `.projects/main@app.openfn.org.yaml` +- It will _checkout_ (expand) that project onto your file system, expanding + workflows and steps to their own files. ## Understanding a Project Structure @@ -118,13 +146,14 @@ There are three key files to understand here: ### project.yaml The project file saves a copy of the whole state of a project as saved in the -app. If you look inside you'll see the workflows represented as0 plain text. +app. If you look inside you'll see the workflows represented as plain text. A project file is named like `@.yaml`. The alias is a local name -used to refer to a particular version of the project. +used to refer to a particular version of the project. The domain comes from the +OpenFn instance the project was downloaded from. -The project file should not be edited locally as changes will be dropped on the -next fetch. +The project file should not be edited locally as any changes will be dropped on +the next fetch. You can fetch as many projects as you like, and each will be saved to its own project.yaml file. @@ -149,21 +178,10 @@ compare/merge them against each other directly. ### openfn.yaml -This is a top-level configuration file - -We call this file system a Workspace (not a project, because technically it -holds multiple projects) - -At any time, one project can be "checked out" or "expanded". That means the the -contents of the `workflows` folder represents that specific project. You cannot -checkout two versions of the same workflow at once. - -- `openfn.yaml` contains settings and metadata about the checked out project -- `.projects` contains a yaml file for each project. This is a local copy of a - remote project. It should always reflect an app's view of a project. You - should generally not edit these files. -- `workflows` contains all the workflows for the checked out project. These are - compatible with the CLI. +This is a top-level configuration file which can mostly be ignored. It is used +by OpenFn tooling to recognise a project root folder. It also holds +configuration options for all local projects, and metadata about the currently +checked out project. ## Aliases @@ -266,8 +284,38 @@ deploy it. ## Sandboxes +The CLI is fully compatible with sandboxes. Treat them like any other project: +fetch them for the first time with their UUID. + +Use the `checkout` command to switch between sandboxes and projects locally. +Remember you can only have a single project checked out at once. The CLI will +warn you if a checkout will cause you to lose any local changes. + +When fetching a sandbox, by default the project alias will be set to the sandbox +name. + +You can merge two projects locally with `openfn merge`, and deploy the resulting +project to the app (you'll likely have to force push the change). This is useful +for conflict resolution. + ## GitHub +You can configure a Project to sync automatically to GitHub. This means that +commit to GitHub will automatically deploy changes to an OpenFn Project; and +pressing Save & Sync in the app will commit back to GitHub. + +Under the hood, GitHub Sync uses the CLI's `pull` and `deploy` commands, +triggered by GitHub Actions, to sync your projects. + +Note that by default, GitHub Sync will use the legacy format, with `state.json`, +`project.yaml` and `config.json` files. When setting up a new GitHub Sync, you +can choose to select the v2 format. V2 sync is only suitable for pulling a +single Project per branch to github, as multiple projects will overwrite the +same `workflows/` folder. + +See [Version Control](/documentation/link-to-GitHub) for more details about +GitHub Sync. + ## Cheatsheet | Command | Description | From c68ee6921bae5a55e575f316fe58841d1d743aa8 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 17:43:16 +0100 Subject: [PATCH 05/14] claude typos --- docs/build-for-developers/cli-sync.md | 28 +++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index 87c62e03919d..dd147132df67 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -4,14 +4,14 @@ sidebar_label: Sync slug: /sync --- -OpefnFn projects are fully portable, which means they can be moved to different +OpenFn projects are fully portable, which means they can be moved to different places. You can create a project in the app, download it to your local machine to develop offline, re-upload it to the app, or even deploy it to a different OpenFn server. -We call this OpenFn Sync, and it's one of the most powerful features affording +We call this OpenFn Sync, and it's one of the most powerful features afforded by OpenFn Projects. ## What is a Project? @@ -19,8 +19,8 @@ by OpenFn Projects. A Project is a set of Workflows which solves, automates or integrates some business function. -A Project lives in the OpenFn app (either on the cloud Saas instance or on a -privately deployed instance ), but it also can exist as files on a file system. +A Project lives in the OpenFn app (either on the cloud SaaS instance or on a +privately deployed instance), but it also can exist as files on a file system. Attached to every Project is some metadata (like a name and a description) and some configuration like Credentials and Collections. @@ -172,7 +172,7 @@ each step to a step.js file. This all lives in the `workflows` directory. Here you can edit files as much as you like, and changes will be tracked when you push/deploy back to the app. -You can only check out on project a time. This is actually great for working +You can only check out one project at a time. This is actually great for working with git, because you can checkout two projects on different branches and compare/merge them against each other directly. @@ -197,7 +197,7 @@ You can pull a project and set the alias at the same time by running openfn project pull --alias dev ``` -This will save the project to `dev@app.openfn.org.yaml` +This will save the project to `dev@app.openfn.org.yaml`. To change the alias, you can simply rename the file. Whatever goes before the `@` will be treated as the alias. @@ -266,7 +266,7 @@ To preview what would change without actually uploading anything, use app (as a JSON structure). You can deploy the checked out project as a new project on the target app by -adding the `--new` command. This is only available if you have superuser +adding the `--new` flag. This is only available if you have superuser privileges on the target instance. You can also deploy the checked out project to another app project by passing @@ -294,7 +294,7 @@ warn you if a checkout will cause you to lose any local changes. When fetching a sandbox, by default the project alias will be set to the sandbox name. -You can merge two projects locally with `openfn merge`, and deploy the resulting +You can merge two projects locally with `openfn project merge`, and deploy the resulting project to the app (you'll likely have to force push the change). This is useful for conflict resolution. @@ -318,12 +318,12 @@ GitHub Sync. ## Cheatsheet -| Command | Description | -| ---------------------------------------- | ------------------------------------------------------------------------------ | ----- | --------------------------------------- | -| `openfn project pull ` | Pull a project from the app for the first time | -| `openfn project pull` | Re-pull the current project | -| `openfn project pull --alias dev` | Pull and set a local alias | -| `openfn project fetch ` | Pull a project from the app for the first time | +| `openfn project pull` | Re-pull the current project | +| `openfn project pull --alias dev` | Pull and set a local alias | +| `openfn project fetch ` | Fetch a project without checking it out | | `openfn project` | List all local projects in the current working folder | | `openfn project checkout ` | Switch to a different local project | | `openfn project deploy` | Deploy checked-out project to the app | From ce39e8c90de44993e997fbb27cbd7f9c8d7debe6 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 17:59:37 +0100 Subject: [PATCH 06/14] thinking about structure --- docs/build-for-developers/cli-sync.md | 28 +- docs/deploy/portability-v1.md | 439 ++++++++++++++++++++++++++ docs/deploy/portability.md | 61 +++- 3 files changed, 498 insertions(+), 30 deletions(-) create mode 100644 docs/deploy/portability-v1.md diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index dd147132df67..fbd5399b1ffb 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -11,8 +11,8 @@ You can create a project in the app, download it to your local machine to develop offline, re-upload it to the app, or even deploy it to a different OpenFn server. -We call this OpenFn Sync, and it's one of the most powerful features afforded -by OpenFn Projects. +We call this OpenFn Sync, and it's one of the most powerful features afforded by +OpenFn Projects. ## What is a Project? @@ -126,7 +126,7 @@ The `project pull` command does three things: - It will _checkout_ (expand) that project onto your file system, expanding workflows and steps to their own files. -## Understanding a Project Structure +## Understanding Project Structure Pulling a project from the app will create a file structure that looks like this: @@ -266,8 +266,8 @@ To preview what would change without actually uploading anything, use app (as a JSON structure). You can deploy the checked out project as a new project on the target app by -adding the `--new` flag. This is only available if you have superuser -privileges on the target instance. +adding the `--new` flag. This is only available if you have superuser privileges +on the target instance. You can also deploy the checked out project to another app project by passing its alias, id or uuid: @@ -294,9 +294,9 @@ warn you if a checkout will cause you to lose any local changes. When fetching a sandbox, by default the project alias will be set to the sandbox name. -You can merge two projects locally with `openfn project merge`, and deploy the resulting -project to the app (you'll likely have to force push the change). This is useful -for conflict resolution. +You can merge two projects locally with `openfn project merge`, and deploy the +resulting project to the app (you'll likely have to force push the change). This +is useful for conflict resolution. ## GitHub @@ -318,12 +318,12 @@ GitHub Sync. ## Cheatsheet -| Command | Description | -| -------------------------------------------- | ------------------------------------------------------------------------------ | -| `openfn project pull ` | Pull a project from the app for the first time | -| `openfn project pull` | Re-pull the current project | -| `openfn project pull --alias dev` | Pull and set a local alias | -| `openfn project fetch ` | Fetch a project without checking it out | +| Command | Description | +| ---------------------------------------- | ------------------------------------------------------------------------------ | +| `openfn project pull ` | Pull a project from the app for the first time | +| `openfn project pull` | Re-pull the current project | +| `openfn project pull --alias dev` | Pull and set a local alias | +| `openfn project fetch ` | Fetch a project without checking it out | | `openfn project` | List all local projects in the current working folder | | `openfn project checkout ` | Switch to a different local project | | `openfn project deploy` | Deploy checked-out project to the app | diff --git a/docs/deploy/portability-v1.md b/docs/deploy/portability-v1.md new file mode 100644 index 000000000000..679872d94e2c --- /dev/null +++ b/docs/deploy/portability-v1.md @@ -0,0 +1,439 @@ +--- +title: Portability v1 (legacy) +--- + +The portability specification allows for the representations of entire workflow +projects "as code"; lets user moves between various deployment pathways (such as +cloud, local, hosted); and proposes a globally-applicable way of defining +workflow automation and system integration rules that might be applied across +workflow-engines/integration platforms across the sector. + +Nothing about the spec _must_ be specific to OpenFn or any one of our individual +products. We envision a future in which software built with Lightning, the +OpenFn Integration Toolkit, and entirely new and different integration/workflow +tools can adopt this specification. + +If you're interested in contributing to the specification, reach out to OpenFn +via the [community forum](https://community.openfn.org), write to us, or suggest +changes by submitting a pull request here. + +:::warning + +This is the legacy version of the OpenFn Portability spec. + +For the latest version, see [Portability](portability) + +::: + +## Projects "as code" + +Entire projects (groups of workflows with their associated triggers, edges, +credentials and jobs) can be represented as code. + +This improves the OpenFn developer experience by (a) allowing workflows to be +built and tested locally; (b) enabling project version control and an audit +trail of project changes; and (c) allowing users to port existing projects +between different instances (i.e., deployments) of Lightning. + +### Directory structure + +Many users keep OpenFn projects in git repositories, and this is a common +structure: + +``` +myProject/ +├── workflow-a/ +│ ├── job-1.js +│ ├── job-2.js +│ └── job-3.js +├── workflow-b/ +│ └── job-4.js +├── project.yaml +├── projectState.json +└── config.json +``` + +:::info Directory Structure + +There are commonly used 3 directory structure for OpenFn projects namely: +standard, production & test, and monorepo. To learn more, please see the OpenFn +[GitHub configuration documentation](/documentation/link-to-GitHub#structuring-your-github-repository). + +::: + +### The project **_spec_** + +The project specification (or "spec") is often saved as a `project.yaml` file. +While most of the spec is written inline, many developers prefer to track their +job bodies in separate `.js` files and they then reference them with a relative +path. + +```yaml +name: openhie-project +description: Some sample +credentials: + jane-smith@test.com-HAPI-FHIR: + owner: jane-smith@test.com + name: HAPI FHIR +workflows: + OpenHIE-Workflow: + name: OpenHIE Workflow + jobs: + FHIR-standard-Data-with-change: + name: FHIR-standard-Data-with-change + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: + path: ./jobs/my-fancy-script.js + + Send-to-OpenHIM-to-route-to-SHR: + name: Send-to-OpenHIM-to-route-to-SHR + adaptor: '@openfn/language-http@latest' + enabled: true + credential: jane-smith@test.com-HAPI-FHIR + body: | + fn(state => { + console.log("hello github integration") + return state + }); + + Notify-CHW-upload-successful: + name: Notify-CHW-upload-successful + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: fn(state => state); + + Notify-CHW-upload-failed: + name: Notify-CHW-upload-failed + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: + path: ./jobs/notify-failure.js + + triggers: + webhook: + type: webhook + edges: + webhook->FHIR-standard-Data-with-change: + source_trigger: webhook + target_job: FHIR-standard-Data-with-change + condition: always + FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR: + source_job: FHIR-standard-Data-with-change + target_job: Send-to-OpenHIM-to-route-to-SHR + condition: on_job_success + Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful: + source_job: Send-to-OpenHIM-to-route-to-SHR + target_job: Notify-CHW-upload-successful + condition: on_job_success + Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed: + source_job: Send-to-OpenHIM-to-route-to-SHR + target_job: Notify-CHW-upload-failed + condition: on_job_failure +``` + +In this spec, you can see the different ways of defining a job's body: + +1. Inline body: Used in the `FHIR-standard-Data-with-change` and + `Send-to-OpenHIM-to-route-to-SHR` jobs. The body is directly written in the + YAML file. + +2. External file reference: Used in both `Notify-CHW-upload-successful` and + `Notify-CHW-upload-failed` jobs. The body is stored in separate files, + referenced by the path key. This allows for better organization of complex + job logic. + +When using file paths: + +- Paths are relative to the location of the `project.yaml` file. +- Ensure that the referenced files exist and contain valid job body code. +- This method is particularly useful for complex jobs or when you want to reuse + job bodies across different projects. + +### The project **_state_** + +The project state is a representation of a particular project as _on a specific +Lightning instance_. It is often saved as `projectState.json` and contains UUIDs +for resources on a particular Lightning deployment. + +```json +{ + "id": "8deff39d-8189-4bd7-9dc7-f9f08e7f2c60", + "name": "openhie-project", + "description": null, + "inserted_at": "2023-08-25T08:57:31", + "updated_at": "2023-08-25T08:57:31", + "scheduled_deletion": null, + "requires_mfa": false, + "project_credentials": { + "jane-smith@test.com-HAPI-FHIR": { + "id": "25f48989-d349-4eb8-99c3-923ebba5b116", + "name": "HAPI FHIR", + "owner": "jane-smith@test.com" + } + }, + "workflows": { + "OpenHIE-Workflow": { + "id": "27ae2937-0959-48b8-a597-b1646aae8c14", + "name": "OpenHIE Workflow", + "jobs": { + "Transform-data-to-FHIR-standard": { + "id": "e44f65bb-5038-4e17-8d93-b63cbe95254a", + "delete": true + }, + "Send-to-OpenHIM-to-route-to-SHR": { + "id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "name": "Send-to-OpenHIM-to-route-to-SHR", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "Notify-CHW-upload-successful": { + "id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b", + "name": "Notify-CHW-upload-successful", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "Notify-CHW-upload-failed": { + "id": "be85df30-0abd-4f8e-be17-501f67e18b8d", + "name": "Notify-CHW-upload-failed", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "FHIR-standard-Data": { + "id": "55016dda-42e3-4ee1-8a9c-24e3f23d42f1", + "delete": true + }, + "FHIR-standard-Data-with-change": { + "id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", + "name": "FHIR-standard-Data-with-change", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + } + }, + "triggers": { + "webhook": { + "id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", + "type": "webhook" + } + }, + "edges": { + "webhook->Transform-data-to-FHIR-standard": { + "id": "b2c7407b-0ae9-4ca5-9d6b-ee624976fa54", + "delete": true + }, + "Transform-data-to-FHIR-standard->Send-to-OpenHIM-to-route-to-SHR": { + "id": "d22ed6f4-26a2-4c85-b261-cc110a6851e6", + "delete": true + }, + "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful": { + "id": "26c12f7f-7806-4008-87cd-6747998f95f4", + "condition": "on_job_success", + "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "source_trigger_id": null, + "target_job_id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b" + }, + "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed": { + "id": "0630ac96-4f67-4de7-8c3d-0bf3f89f80d9", + "condition": "on_job_failure", + "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "source_trigger_id": null, + "target_job_id": "be85df30-0abd-4f8e-be17-501f67e18b8d" + }, + "webhook->FHIR-standard-Data": { + "id": "5ce3a8ed-b9eb-464a-a2cd-ba55adc393c2", + "delete": true + }, + "FHIR-standard-Data->Send-to-OpenHIM-to-route-to-SHR": { + "id": "5f459cd9-2882-4a61-a2cc-ec45e58d4837", + "delete": true + }, + "webhook->FHIR-standard-Data-with-change": { + "id": "75e7f7d8-274b-410d-9600-730bbd535229", + "condition": "always", + "source_job_id": null, + "source_trigger_id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", + "target_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe" + }, + "FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR": { + "id": "1e5ba385-2c49-4241-8cd2-042c99a810ec", + "condition": "on_job_success", + "source_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", + "source_trigger_id": null, + "target_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32" + } + } + } + } +} +``` + +## Using the CLI interact with projects + +The project spec and project state can be used for a variety of reasons, e.g. +one could generate the state and spec as backups of the project or one could +generate these files and use them for auditing and record keeping, etc. The +OpenFn [CLI](https://github.com/OpenFn/kit/tree/main/packages/cli) comes with +commands that can be used to pull project configurations down from a running +Lightning server, and to deploy or push updates to existing projects on a +Lightning server. To learn more about automated version control via pull and +deploy, head over to our [Version Control](../manage-projects/link-to-gh.md) +docs. + +:::info Don't have the CLI yet? + +Install it by running `npm install -g @openfn/cli` + +::: + +Before using the CLI, configure it either by passing in environment variables: + +``` +OPENFN_ENDPOINT=https://app.openfn.org +OPENFN_API_KEY=yourSecretApiToken +``` + +Or through a `config.json` file: + +```json +{ + // Required, can be overridden or set with `OPENFN_API_KEY` env var + "apiKey": "***", + + // Optional: can be set using the -p, defaults to project.yaml + "specPath": "project.yaml", + + // Optional: can be set using -s, defaults to .state.json + "statePath": ".state.json", + + // Optional: defaults to OpenFn.org's API, can be overridden or set with + // `OPENFN_ENDPOINT` env var + "endpoint": "https://app.openfn.org" +} +``` + +More details on the CLI can be found +[here](https://github.com/OpenFn/kit/tree/main/packages/cli#basic-usage). + +### `openfn pull` to generate spec & state + +To generate the spec and state files for an existing project, use: + +```sh +openfn pull {YOUR-PROJECT-UUID} -c ./config.json +``` + +This command will save (or overwrite) a project spec and state file based on the +path you've set in your configuration. + +### `openfn deploy` to create new projects + +To deploy a new project to a Lightning instance from a project spec (without a +project state) file use: + +```sh +openfn deploy -c config.json +``` + +### `openfn deploy` to update existing projects + +With a valid project state defined in your `config.json`, the same +`openfn deploy` command will beam up your changes as described by a difference +between your project spec and what's found on the server. + +```sh +openfn deploy -c config.json +Checking https://demo.openfn.org/api/provision/4adf2644-ed4e-4f97-a24c-ab35b3cb1efa for existing project. +Project found. +[CLI] ♦ Changes: + { + workflows: [ + { + jobs: [ + { +- body: "fn(state => {\n console.log(\"ok\")\n return state\n});" ++ body: "fn(state => {\n console.log(\"some changes here!\")\n return state\n});\n" + } + ... + ... + ... + ] + } + ] + } + +? Deploy? yes +[CLI] ♦ Deployed. +``` + +## Getting Help with the cli + +The cli package comes with an inbuilt `help`. Adding `--help` to a command such +as `openfn deploy --help` will result in a help message describing the command +and the options available when using this command. See an example below + +```sh +openfn deploy --help +openfn deploy + +Deploy a project's config to a remote Lightning instance + +Options: + --version Show version number [boolean] + --help Show help [boolean] + -c, --config, --config-path The location of your config file [default: "./.config.json"] + --no-confirm Skip confirmation prompts (e.g. 'Are you sure?') [boolean] + --describe Downloads the project yaml from the specified instance [boolean] + -l, --log Set the log level [string] + --log-json Output all logs as JSON objects [boolean] + -p, --project-path The location of your project.yaml file [string] + -s, --state-path Path to the state file +``` + +## Troubleshooting + +This section covers solutions to some errors you might come across when using +OpenFn pull or deploy in your projects. + +### Extraneous Workflow ID + +#### Description + +This error occurs when you run `openfn deploy` and there is a mismatch between +between IDs of workflows in your projectSpec and your OpenFn instance. When this +occurs, the error will be written out in an error object as shown below: + +``` +[CLI] ✘ Failed to deploy project openfn-data-buffers-prototype: +{ + "errors": { + "workflows": { + "1-ingest-messages": { + "base": [ + "extraneous parameters: workflow_id" + ] + }, + "2-calculate-indicators": { + "base": [ + "extraneous parameters: workflow_id" + ] + } + } + } +``` + +#### Solution + +Run `openfn pull` to update your local instance and keep IDs in sync, +incorporate your changes and run `openfn deploy` again. + +## Other Versions + +- [Portability Spec v2](portability-versions#v2) +- [Portability Spec v1](portability-versions#v1) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 10f9d2ac8973..1c48ecfb54d0 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -2,30 +2,58 @@ title: Portability --- -## Intent - The portability specification allows for the representations of entire workflow -projects "as code", lets user move between various deployment pathways (cloud, -local, DIY, etc.) and proposes a globally-applicable way of **_specifying -workflow automation_** and **_systems integration_** that might be applied -across workflow-engines/integration platforms across the sector. Nothing about -the spec _must_ be specific to OpenFn or any one of our individual products. We -envision a future in which software built with Lightning, the OpenFn Integration -Toolkit, and entirely new and different integration/workflow tools can adopt -this specification. +projects "as code"; lets user moves between various deployment pathways (such as +cloud, local, hosted); and proposes a globally-applicable way of defining +workflow automation and system integration rules that might be applied across +workflow-engines/integration platforms across the sector. + +Nothing about the spec _must_ be specific to OpenFn or any one of our individual +products. We envision a future in which software built with Lightning, the +OpenFn Integration Toolkit, and entirely new and different integration/workflow +tools can adopt this specification. If you're interested in contributing to the specification, reach out to OpenFn via the [community forum](https://community.openfn.org), write to us, or suggest changes by submitting a pull request here. +:::info + +This document describes v2 of the OpenFn deployment spec, in-line with +[OpenFn Sync](/documentation/sync). + +For the v1 spec (using state, project and config files), see +[Legacy Portability](./portability-v1). + +::: + ## Projects "as code" Entire projects (groups of workflows with their associated triggers, edges, -credentials and jobs) can be represented as code. This improves the OpenFn -developer experience by (a) allowing workflows to be built and tested locally; -(b) enabling project version control and an audit trail of project changes; and -(c) allowing users to port existing projects between different instances (i.e., -deployments) of Lightning. +credentials and jobs) can be represented as code. + +This improves the OpenFn developer experience by: + +1. Allowing workflows to be built and tested locally +2. Enabling project version control and an audit trail of project changes +3. Allowing users to port existing projects between different instances (i.e., + deployments) of Lightning. + +# Joe's Thoughts + +What should go in this file? + +The Sync document is the best equivalent. Should I just reference that? + +This document talks about a portability "spec". But I don't really see that +explained anywhere. Should I rewrite that stuff into a higher level manifesto? + +There's also already a v1 and v2 spec. So actually what I'm doing now is a v3? + +I should chat with Taylor about what to do here. + +I am super unconfortable having multiple versions of the sync workflow on the +site ### Directory structure @@ -395,7 +423,7 @@ OpenFn pull or deploy in your projects. ### Extraneous Workflow ID -#### Description +#### Description This error occurs when you run `openfn deploy` and there is a mismatch between between IDs of workflows in your projectSpec and your OpenFn instance. When this @@ -419,6 +447,7 @@ occurs, the error will be written out in an error object as shown below: } } ``` + #### Solution Run `openfn pull` to update your local instance and keep IDs in sync, From 003cb63343e0f475f3336bbbdd54a2eca5f3c6f0 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 18:11:22 +0100 Subject: [PATCH 07/14] fiddling with sync --- docs/manage-projects/link-to-gh.md | 38 +++++++++++++++--------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/manage-projects/link-to-gh.md b/docs/manage-projects/link-to-gh.md index 92a8204f03ec..f919be00199b 100644 --- a/docs/manage-projects/link-to-gh.md +++ b/docs/manage-projects/link-to-gh.md @@ -1,31 +1,31 @@ --- -title: Version Control -sidebar_label: Version Control (GitHub Sync) +title: GitHub Sync +sidebar_label: GitHub Sync slug: /link-to-GitHub --- -The version control (GitHub Sync) feature lets users track and manage changes to -their OpenFn projects in GitHub. GitHub Sync enables a 2-way sync between your -OpenFn project and your GitHub repository. By 2-way sync, we mean that you can -**sync changes made to your project on OpenFn to GitHub**, and you can **deploy -changes you've made to your project on GitHub to OpenFn**. +GitHub Sync enables two-way sync between OpenFn project and your GitHub +repository. + +This means changes made to a Project in OpenFn can commit back to your GitHub +repo; and commits pushed to GitHub can update the Project in your OpenFn App. :::info For Cloud Hosted OpenFn Users GitHub Sync is only available in projects that are subscribed to Core, Growth, -Scale or Custom plans on Cloud Hosted OpenFn. Users can however connect their -OpenFn account to GitHub by authenticating OpenFn to access their GitHub account -by navigating to `User Profile` page and clicking ` Connect your GitHub -Account". +Scale or Custom plans on Cloud Hosted OpenFn. + +Users can however connect their OpenFn account to GitHub by authenticating +OpenFn to access their GitHub account by navigating to `User Profile` page and +clicking ` Connect your GitHub Account". ::: -### Configuring your project to use GitHub Sync +### Configuring your Project to use GitHub Sync -Users are able to configure their projects to have access to one or more -repositories on GitHub. To enable sync, the OpenFn project requires a repository -where a GitHub OpenFn application is installed and users are required to have -administrative access to the repositiory. +Users can configure their projects to access one or more repositories on GitHub. +Users must have admin access to the GitHub Repo to ensure the OpenFn application +is installed. To configure your project to use Github sync, follow the following steps: @@ -285,8 +285,6 @@ your-git-monorepo :::tip A sync in time, saves nine -#### Syncing Changes from OpenFn to GitHub - When you sync changes from OpenFn to GitHub, the `projectSpec.yaml` file in your repository will be updated with the changes made to the project in OpenFn. For a project with a directory structure that uses relative paths for job code, OpenFn @@ -298,7 +296,9 @@ When you keep job code in relative file paths, ensure to update the `projectSpec.yaml` file based on changes to the files or paths in your project repository. A GitHub action is automatically triggered to push changes to OpenFn ensuring that future syncs are not affected. Changes can include adding, -renaming, deleting a file or updating a file path. ::: +renaming, deleting a file or updating a file path. + +::: ## Troubleshooting From d57d90f6e89ba949ae89de7e1be28ce14ba934b0 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 12:13:33 +0100 Subject: [PATCH 08/14] remove notes --- docs/deploy/portability.md | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 1c48ecfb54d0..4a89c08988b3 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -39,22 +39,6 @@ This improves the OpenFn developer experience by: 3. Allowing users to port existing projects between different instances (i.e., deployments) of Lightning. -# Joe's Thoughts - -What should go in this file? - -The Sync document is the best equivalent. Should I just reference that? - -This document talks about a portability "spec". But I don't really see that -explained anywhere. Should I rewrite that stuff into a higher level manifesto? - -There's also already a v1 and v2 spec. So actually what I'm doing now is a v3? - -I should chat with Taylor about what to do here. - -I am super unconfortable having multiple versions of the sync workflow on the -site - ### Directory structure Many users keep OpenFn projects in git repositories, and this is a common From 1d3d5ff984114b95ca6d22e70ba2cde687b7bb7d Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:33:24 +0100 Subject: [PATCH 09/14] rollback portability stuff --- docs/deploy/portability.md | 45 ++++++++++++++------------------------ 1 file changed, 16 insertions(+), 29 deletions(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 4a89c08988b3..10f9d2ac8973 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -2,42 +2,30 @@ title: Portability --- -The portability specification allows for the representations of entire workflow -projects "as code"; lets user moves between various deployment pathways (such as -cloud, local, hosted); and proposes a globally-applicable way of defining -workflow automation and system integration rules that might be applied across -workflow-engines/integration platforms across the sector. +## Intent -Nothing about the spec _must_ be specific to OpenFn or any one of our individual -products. We envision a future in which software built with Lightning, the -OpenFn Integration Toolkit, and entirely new and different integration/workflow -tools can adopt this specification. +The portability specification allows for the representations of entire workflow +projects "as code", lets user move between various deployment pathways (cloud, +local, DIY, etc.) and proposes a globally-applicable way of **_specifying +workflow automation_** and **_systems integration_** that might be applied +across workflow-engines/integration platforms across the sector. Nothing about +the spec _must_ be specific to OpenFn or any one of our individual products. We +envision a future in which software built with Lightning, the OpenFn Integration +Toolkit, and entirely new and different integration/workflow tools can adopt +this specification. If you're interested in contributing to the specification, reach out to OpenFn via the [community forum](https://community.openfn.org), write to us, or suggest changes by submitting a pull request here. -:::info - -This document describes v2 of the OpenFn deployment spec, in-line with -[OpenFn Sync](/documentation/sync). - -For the v1 spec (using state, project and config files), see -[Legacy Portability](./portability-v1). - -::: - ## Projects "as code" Entire projects (groups of workflows with their associated triggers, edges, -credentials and jobs) can be represented as code. - -This improves the OpenFn developer experience by: - -1. Allowing workflows to be built and tested locally -2. Enabling project version control and an audit trail of project changes -3. Allowing users to port existing projects between different instances (i.e., - deployments) of Lightning. +credentials and jobs) can be represented as code. This improves the OpenFn +developer experience by (a) allowing workflows to be built and tested locally; +(b) enabling project version control and an audit trail of project changes; and +(c) allowing users to port existing projects between different instances (i.e., +deployments) of Lightning. ### Directory structure @@ -407,7 +395,7 @@ OpenFn pull or deploy in your projects. ### Extraneous Workflow ID -#### Description +#### Description This error occurs when you run `openfn deploy` and there is a mismatch between between IDs of workflows in your projectSpec and your OpenFn instance. When this @@ -431,7 +419,6 @@ occurs, the error will be written out in an error object as shown below: } } ``` - #### Solution Run `openfn pull` to update your local instance and keep IDs in sync, From 266e1965710b744460788fb68d26870fc9d92144 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:56:47 +0100 Subject: [PATCH 10/14] explain workflow.yaml --- docs/build-for-developers/cli-sync.md | 155 ++++++++++++++++---------- 1 file changed, 98 insertions(+), 57 deletions(-) diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index fbd5399b1ffb..63df02035eb2 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -55,6 +55,104 @@ Not all artifacts of a Project are included in a sync. Generally we sync the project's workflows and some of its options. But we do not sync any associated data, credential values, usage history or AI sessions. +## Project Structure + +OpenFn Sync writes a Project to the file system using a number of conventions. +Whether using the CLI or GitHub Sync, a Project has the following structure + +``` +├── openfn.yaml +├── .projects +│ ├── main@app.openfn.org.yaml +└── workflows + ├── my-workflow + │ ├── my-workflow.yaml + │ ├── my-step.js +``` + +Briefly, these files are: + +- `openfn.yaml` declares this folder to be an OpenFn Project, and contains + metadata and settings +- The `projects` folder contains a complete YAML representation of each Project +- The `workflows` folder shows the contents of the project - the steps and edges + and so on. + +Let's look at this structure in a bit more detail. + +### project.yaml + +The project file saves a copy of the whole state of a project as saved in the +app. If you look inside you'll see the workflows represented as plain text. + +A project file is named like `@.yaml`. The alias is a local name +used to refer to a particular version of the project. The domain comes from the +OpenFn instance the project was downloaded from. + +The project file should not be edited locally as any changes will be dropped on +the next fetch. + +You can fetch as many projects as you like, and each will be saved to its own +project.yaml file. + +The `.projects` folder can and should be committed to source control. + +### workflows + +Having your whole project inside a single file isn't actually a great way to +read or edit workflows. So the CLI can "checkout" or "expand" a project file +onto the file system. + +Checking out is the process of writing each workflow to a workflow.yaml file and +each step to a step.js file. This all lives in the `workflows` directory. + +Here you can edit files as much as you like, and changes will be tracked when +you push/deploy back to the app. + +You can only check out one project at a time. This is actually great for working +with git, because you can checkout two projects on different branches and +compare/merge them against each other directly. + +### workflow.yaml + +A workflow.yaml file defines the steps of a workflow and the edges which connect +them. + +``` +id: my-workflow +name: My Workflow +start: webhook +steps: + - id: my-step + name: My Step + adaptor: '@openfn/language-http@7.2.9' + expression: ./my-step.js + - id: webhook + type: webhook + enabled: true + next: + my-step: + disabled: false + condition: always +``` + +The `next` key on each step defines any downstream edges for that step - ie, the +steps to execute next. In the example above, the `webbhook` step is executed +first (determined by the `start` key), and it defines a single edge to +`my-step`, which always executes. + +The actual code for each step lives in its own .js file. You can modify the code +freely and sync it back to the server any time. If you want to rename a step, +make sure to to update the file name of the step and the path in the +`expression` key in `workflow.yaml`. + +### openfn.yaml + +This is a top-level configuration file which can mostly be ignored. It is used +by OpenFn tooling to recognise a project root folder. It also holds +configuration options for all local projects, and metadata about the currently +checked out project. + ## Authorization Before you use the CLI to fetch anything from the app, you'll need to provide @@ -126,63 +224,6 @@ The `project pull` command does three things: - It will _checkout_ (expand) that project onto your file system, expanding workflows and steps to their own files. -## Understanding Project Structure - -Pulling a project from the app will create a file structure that looks like -this: - -``` -├── openfn.yaml -├── .projects -│ ├── main@app.staging.openfn.org.yaml -└── workflows - ├── my-workflow - │ ├── my-workflow.yaml - │ ├── job.js -``` - -There are three key files to understand here: - -### project.yaml - -The project file saves a copy of the whole state of a project as saved in the -app. If you look inside you'll see the workflows represented as plain text. - -A project file is named like `@.yaml`. The alias is a local name -used to refer to a particular version of the project. The domain comes from the -OpenFn instance the project was downloaded from. - -The project file should not be edited locally as any changes will be dropped on -the next fetch. - -You can fetch as many projects as you like, and each will be saved to its own -project.yaml file. - -The `.projects` folder can and should be committed to source control. - -### workflows - -Having your whole project inside a single file isn't actually a great way to -read or edit workflows. So the CLI can "checkout" or "expand" a project file -onto the file system. - -Checking out is the process of writing each workflow to a workflow.yaml file and -each step to a step.js file. This all lives in the `workflows` directory. - -Here you can edit files as much as you like, and changes will be tracked when -you push/deploy back to the app. - -You can only check out one project at a time. This is actually great for working -with git, because you can checkout two projects on different branches and -compare/merge them against each other directly. - -### openfn.yaml - -This is a top-level configuration file which can mostly be ignored. It is used -by OpenFn tooling to recognise a project root folder. It also holds -configuration options for all local projects, and metadata about the currently -checked out project. - ## Aliases Rather than identify a local project with a long-winded UUID or id string, you From 613684dd118828d1a99e1fba6b34dfb22eb17ccf Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:57:09 +0100 Subject: [PATCH 11/14] tidy --- docs/deploy/portability-v1.md | 439 ---------------------------------- 1 file changed, 439 deletions(-) delete mode 100644 docs/deploy/portability-v1.md diff --git a/docs/deploy/portability-v1.md b/docs/deploy/portability-v1.md deleted file mode 100644 index 679872d94e2c..000000000000 --- a/docs/deploy/portability-v1.md +++ /dev/null @@ -1,439 +0,0 @@ ---- -title: Portability v1 (legacy) ---- - -The portability specification allows for the representations of entire workflow -projects "as code"; lets user moves between various deployment pathways (such as -cloud, local, hosted); and proposes a globally-applicable way of defining -workflow automation and system integration rules that might be applied across -workflow-engines/integration platforms across the sector. - -Nothing about the spec _must_ be specific to OpenFn or any one of our individual -products. We envision a future in which software built with Lightning, the -OpenFn Integration Toolkit, and entirely new and different integration/workflow -tools can adopt this specification. - -If you're interested in contributing to the specification, reach out to OpenFn -via the [community forum](https://community.openfn.org), write to us, or suggest -changes by submitting a pull request here. - -:::warning - -This is the legacy version of the OpenFn Portability spec. - -For the latest version, see [Portability](portability) - -::: - -## Projects "as code" - -Entire projects (groups of workflows with their associated triggers, edges, -credentials and jobs) can be represented as code. - -This improves the OpenFn developer experience by (a) allowing workflows to be -built and tested locally; (b) enabling project version control and an audit -trail of project changes; and (c) allowing users to port existing projects -between different instances (i.e., deployments) of Lightning. - -### Directory structure - -Many users keep OpenFn projects in git repositories, and this is a common -structure: - -``` -myProject/ -├── workflow-a/ -│ ├── job-1.js -│ ├── job-2.js -│ └── job-3.js -├── workflow-b/ -│ └── job-4.js -├── project.yaml -├── projectState.json -└── config.json -``` - -:::info Directory Structure - -There are commonly used 3 directory structure for OpenFn projects namely: -standard, production & test, and monorepo. To learn more, please see the OpenFn -[GitHub configuration documentation](/documentation/link-to-GitHub#structuring-your-github-repository). - -::: - -### The project **_spec_** - -The project specification (or "spec") is often saved as a `project.yaml` file. -While most of the spec is written inline, many developers prefer to track their -job bodies in separate `.js` files and they then reference them with a relative -path. - -```yaml -name: openhie-project -description: Some sample -credentials: - jane-smith@test.com-HAPI-FHIR: - owner: jane-smith@test.com - name: HAPI FHIR -workflows: - OpenHIE-Workflow: - name: OpenHIE Workflow - jobs: - FHIR-standard-Data-with-change: - name: FHIR-standard-Data-with-change - adaptor: '@openfn/language-http@latest' - enabled: true - credential: null - body: - path: ./jobs/my-fancy-script.js - - Send-to-OpenHIM-to-route-to-SHR: - name: Send-to-OpenHIM-to-route-to-SHR - adaptor: '@openfn/language-http@latest' - enabled: true - credential: jane-smith@test.com-HAPI-FHIR - body: | - fn(state => { - console.log("hello github integration") - return state - }); - - Notify-CHW-upload-successful: - name: Notify-CHW-upload-successful - adaptor: '@openfn/language-http@latest' - enabled: true - credential: null - body: fn(state => state); - - Notify-CHW-upload-failed: - name: Notify-CHW-upload-failed - adaptor: '@openfn/language-http@latest' - enabled: true - credential: null - body: - path: ./jobs/notify-failure.js - - triggers: - webhook: - type: webhook - edges: - webhook->FHIR-standard-Data-with-change: - source_trigger: webhook - target_job: FHIR-standard-Data-with-change - condition: always - FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR: - source_job: FHIR-standard-Data-with-change - target_job: Send-to-OpenHIM-to-route-to-SHR - condition: on_job_success - Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful: - source_job: Send-to-OpenHIM-to-route-to-SHR - target_job: Notify-CHW-upload-successful - condition: on_job_success - Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed: - source_job: Send-to-OpenHIM-to-route-to-SHR - target_job: Notify-CHW-upload-failed - condition: on_job_failure -``` - -In this spec, you can see the different ways of defining a job's body: - -1. Inline body: Used in the `FHIR-standard-Data-with-change` and - `Send-to-OpenHIM-to-route-to-SHR` jobs. The body is directly written in the - YAML file. - -2. External file reference: Used in both `Notify-CHW-upload-successful` and - `Notify-CHW-upload-failed` jobs. The body is stored in separate files, - referenced by the path key. This allows for better organization of complex - job logic. - -When using file paths: - -- Paths are relative to the location of the `project.yaml` file. -- Ensure that the referenced files exist and contain valid job body code. -- This method is particularly useful for complex jobs or when you want to reuse - job bodies across different projects. - -### The project **_state_** - -The project state is a representation of a particular project as _on a specific -Lightning instance_. It is often saved as `projectState.json` and contains UUIDs -for resources on a particular Lightning deployment. - -```json -{ - "id": "8deff39d-8189-4bd7-9dc7-f9f08e7f2c60", - "name": "openhie-project", - "description": null, - "inserted_at": "2023-08-25T08:57:31", - "updated_at": "2023-08-25T08:57:31", - "scheduled_deletion": null, - "requires_mfa": false, - "project_credentials": { - "jane-smith@test.com-HAPI-FHIR": { - "id": "25f48989-d349-4eb8-99c3-923ebba5b116", - "name": "HAPI FHIR", - "owner": "jane-smith@test.com" - } - }, - "workflows": { - "OpenHIE-Workflow": { - "id": "27ae2937-0959-48b8-a597-b1646aae8c14", - "name": "OpenHIE Workflow", - "jobs": { - "Transform-data-to-FHIR-standard": { - "id": "e44f65bb-5038-4e17-8d93-b63cbe95254a", - "delete": true - }, - "Send-to-OpenHIM-to-route-to-SHR": { - "id": "977b87ff-f347-42b5-832f-6ae2ca726f32", - "name": "Send-to-OpenHIM-to-route-to-SHR", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - }, - "Notify-CHW-upload-successful": { - "id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b", - "name": "Notify-CHW-upload-successful", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - }, - "Notify-CHW-upload-failed": { - "id": "be85df30-0abd-4f8e-be17-501f67e18b8d", - "name": "Notify-CHW-upload-failed", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - }, - "FHIR-standard-Data": { - "id": "55016dda-42e3-4ee1-8a9c-24e3f23d42f1", - "delete": true - }, - "FHIR-standard-Data-with-change": { - "id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", - "name": "FHIR-standard-Data-with-change", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - } - }, - "triggers": { - "webhook": { - "id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", - "type": "webhook" - } - }, - "edges": { - "webhook->Transform-data-to-FHIR-standard": { - "id": "b2c7407b-0ae9-4ca5-9d6b-ee624976fa54", - "delete": true - }, - "Transform-data-to-FHIR-standard->Send-to-OpenHIM-to-route-to-SHR": { - "id": "d22ed6f4-26a2-4c85-b261-cc110a6851e6", - "delete": true - }, - "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful": { - "id": "26c12f7f-7806-4008-87cd-6747998f95f4", - "condition": "on_job_success", - "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", - "source_trigger_id": null, - "target_job_id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b" - }, - "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed": { - "id": "0630ac96-4f67-4de7-8c3d-0bf3f89f80d9", - "condition": "on_job_failure", - "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", - "source_trigger_id": null, - "target_job_id": "be85df30-0abd-4f8e-be17-501f67e18b8d" - }, - "webhook->FHIR-standard-Data": { - "id": "5ce3a8ed-b9eb-464a-a2cd-ba55adc393c2", - "delete": true - }, - "FHIR-standard-Data->Send-to-OpenHIM-to-route-to-SHR": { - "id": "5f459cd9-2882-4a61-a2cc-ec45e58d4837", - "delete": true - }, - "webhook->FHIR-standard-Data-with-change": { - "id": "75e7f7d8-274b-410d-9600-730bbd535229", - "condition": "always", - "source_job_id": null, - "source_trigger_id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", - "target_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe" - }, - "FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR": { - "id": "1e5ba385-2c49-4241-8cd2-042c99a810ec", - "condition": "on_job_success", - "source_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", - "source_trigger_id": null, - "target_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32" - } - } - } - } -} -``` - -## Using the CLI interact with projects - -The project spec and project state can be used for a variety of reasons, e.g. -one could generate the state and spec as backups of the project or one could -generate these files and use them for auditing and record keeping, etc. The -OpenFn [CLI](https://github.com/OpenFn/kit/tree/main/packages/cli) comes with -commands that can be used to pull project configurations down from a running -Lightning server, and to deploy or push updates to existing projects on a -Lightning server. To learn more about automated version control via pull and -deploy, head over to our [Version Control](../manage-projects/link-to-gh.md) -docs. - -:::info Don't have the CLI yet? - -Install it by running `npm install -g @openfn/cli` - -::: - -Before using the CLI, configure it either by passing in environment variables: - -``` -OPENFN_ENDPOINT=https://app.openfn.org -OPENFN_API_KEY=yourSecretApiToken -``` - -Or through a `config.json` file: - -```json -{ - // Required, can be overridden or set with `OPENFN_API_KEY` env var - "apiKey": "***", - - // Optional: can be set using the -p, defaults to project.yaml - "specPath": "project.yaml", - - // Optional: can be set using -s, defaults to .state.json - "statePath": ".state.json", - - // Optional: defaults to OpenFn.org's API, can be overridden or set with - // `OPENFN_ENDPOINT` env var - "endpoint": "https://app.openfn.org" -} -``` - -More details on the CLI can be found -[here](https://github.com/OpenFn/kit/tree/main/packages/cli#basic-usage). - -### `openfn pull` to generate spec & state - -To generate the spec and state files for an existing project, use: - -```sh -openfn pull {YOUR-PROJECT-UUID} -c ./config.json -``` - -This command will save (or overwrite) a project spec and state file based on the -path you've set in your configuration. - -### `openfn deploy` to create new projects - -To deploy a new project to a Lightning instance from a project spec (without a -project state) file use: - -```sh -openfn deploy -c config.json -``` - -### `openfn deploy` to update existing projects - -With a valid project state defined in your `config.json`, the same -`openfn deploy` command will beam up your changes as described by a difference -between your project spec and what's found on the server. - -```sh -openfn deploy -c config.json -Checking https://demo.openfn.org/api/provision/4adf2644-ed4e-4f97-a24c-ab35b3cb1efa for existing project. -Project found. -[CLI] ♦ Changes: - { - workflows: [ - { - jobs: [ - { -- body: "fn(state => {\n console.log(\"ok\")\n return state\n});" -+ body: "fn(state => {\n console.log(\"some changes here!\")\n return state\n});\n" - } - ... - ... - ... - ] - } - ] - } - -? Deploy? yes -[CLI] ♦ Deployed. -``` - -## Getting Help with the cli - -The cli package comes with an inbuilt `help`. Adding `--help` to a command such -as `openfn deploy --help` will result in a help message describing the command -and the options available when using this command. See an example below - -```sh -openfn deploy --help -openfn deploy - -Deploy a project's config to a remote Lightning instance - -Options: - --version Show version number [boolean] - --help Show help [boolean] - -c, --config, --config-path The location of your config file [default: "./.config.json"] - --no-confirm Skip confirmation prompts (e.g. 'Are you sure?') [boolean] - --describe Downloads the project yaml from the specified instance [boolean] - -l, --log Set the log level [string] - --log-json Output all logs as JSON objects [boolean] - -p, --project-path The location of your project.yaml file [string] - -s, --state-path Path to the state file -``` - -## Troubleshooting - -This section covers solutions to some errors you might come across when using -OpenFn pull or deploy in your projects. - -### Extraneous Workflow ID - -#### Description - -This error occurs when you run `openfn deploy` and there is a mismatch between -between IDs of workflows in your projectSpec and your OpenFn instance. When this -occurs, the error will be written out in an error object as shown below: - -``` -[CLI] ✘ Failed to deploy project openfn-data-buffers-prototype: -{ - "errors": { - "workflows": { - "1-ingest-messages": { - "base": [ - "extraneous parameters: workflow_id" - ] - }, - "2-calculate-indicators": { - "base": [ - "extraneous parameters: workflow_id" - ] - } - } - } -``` - -#### Solution - -Run `openfn pull` to update your local instance and keep IDs in sync, -incorporate your changes and run `openfn deploy` again. - -## Other Versions - -- [Portability Spec v2](portability-versions#v2) -- [Portability Spec v1](portability-versions#v1) From bd1378544690b93e5d8d75b1ed5db9a8b18d76ff Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:59:52 +0100 Subject: [PATCH 12/14] add legacy note to portability docs --- docs/deploy/portability.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 10f9d2ac8973..e8e40608c02f 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -268,6 +268,16 @@ for resources on a particular Lightning deployment. ## Using the CLI interact with projects +:::warning + +This document describes the legacy OpenFn Sync format used by older versions of +the CLI and GitHub Sync. + +See the [CLI Sync](/documentation/sync) docs for the best experience syncing and +deploying projects. + +::: + The project spec and project state can be used for a variety of reasons, e.g. one could generate the state and spec as backups of the project or one could generate these files and use them for auditing and record keeping, etc. The @@ -395,7 +405,7 @@ OpenFn pull or deploy in your projects. ### Extraneous Workflow ID -#### Description +#### Description This error occurs when you run `openfn deploy` and there is a mismatch between between IDs of workflows in your projectSpec and your OpenFn instance. When this @@ -419,6 +429,7 @@ occurs, the error will be written out in an error object as shown below: } } ``` + #### Solution Run `openfn pull` to update your local instance and keep IDs in sync, From d2abf7508d6012d9793df4f66462a020f9602217 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 15:28:57 +0100 Subject: [PATCH 13/14] nod to v2 format --- docs/manage-projects/link-to-gh.md | 134 +++++++++++------------------ 1 file changed, 52 insertions(+), 82 deletions(-) diff --git a/docs/manage-projects/link-to-gh.md b/docs/manage-projects/link-to-gh.md index f919be00199b..8f36c8b3b0ad 100644 --- a/docs/manage-projects/link-to-gh.md +++ b/docs/manage-projects/link-to-gh.md @@ -13,11 +13,7 @@ repo; and commits pushed to GitHub can update the Project in your OpenFn App. :::info For Cloud Hosted OpenFn Users GitHub Sync is only available in projects that are subscribed to Core, Growth, -Scale or Custom plans on Cloud Hosted OpenFn. - -Users can however connect their OpenFn account to GitHub by authenticating -OpenFn to access their GitHub account by navigating to `User Profile` page and -clicking ` Connect your GitHub Account". +Scale or Custom plans. ::: @@ -194,8 +190,59 @@ Learn more about relative paths and directory structure in ::: +### Using Sync v2 + +By default, GitHub Sync will use the legacy folder structure to represent your +project on GitHub. That folder structure is explained below - it'll create +`config.json`, `state.json` and `project.yaml` files to represent your Project +in your git repo. + +You can instead choose to use the v2 sync format, as described in the +[Sync](/documentation/sync) pages. This will automatically "expand" your +workflows and steps into files to be easily read and written, and provides a +much better developer experience. + +This v2 style will be made the default means of syncing Projects soon. + +You can also create an empty `openfn.yaml` file in an already connected repo and +the next sync will generate the v2 file structures. + +:::warning + +In Sync v1, you can have multiple 2-way syncs existing on the same branch in a +single repo. That's because each project creates its own set of sync artifacts +(config.json, project.yaml and state.json). Usually you'd want to do this to +sync your production and staging projects, or multiple sandboxes, to the same +GitHub repo. + +This doesn't work with the new sync protocol, because the new sync shares a +`workflows` folder. So each time GitHub pulls from your project, it'll overwrite +`workflows` and wipe out state from your other projects. + +To do this in v2 Sync, you can: + +- Maintain one two-way sync per branch. Each Sandbox maintains its own GitHub + Sync to a different branch in your repo. This works great because you can + compare differences between your sandbox and main project by comparing the + branches on git. +- Connect many Projects to a branch so long as they only sync one-way. This + works in a production environment where one Project is replicated over several + deployments, so committing to GitHub will trigger an update to all connected + Projects. This works so long as you can promise that no user will Save & Sync + back from the production projects + +::: + ## What is in your GitHub Repository? +:::info + +These docs describe the legacy format of GitHub Sync. The latest format is +described in the [CLI Sync](/documentation/sync) pages, and will be used by +default soon. + +::: + When you initiate a connection between OpenFn and your GitHub repository, a config.json file containing reference to your project spec and project state files, and the endpoint of your OpenFn deployment is automatically created in @@ -223,83 +270,6 @@ spec and project state. } ``` -## Structuring OpenFn projects in git repositories - -There are three common patterns used to structure OpenFn projects inside git -repositories. See them below: - -### Standard - -Use this approach if you've got one OpenFn project connected to one git -repository. - -``` -your-git-repo -├── config.json -├── projectState.json -└── projectSpec.yaml -``` - -### Production & Test - -Use this approach if you've got two OpenFn projects that use the _same -worklows_. Here, you're connecting two projects (prod and test) to a single git -repo and a single `project.yaml` file. - -This will allow you to keep two projects in sync when changes are merged from -one branch to another. You might choose to sync: - -- Your production project with the `main` branch -- Your test project with the `staging` branch - -After a merge, your repo would look like this: - -``` -your-git-repo -├── projectSpec.yaml ## works both -│ -├── prod-config.json -├── prod-projectState.json -│ -├── test-config.json -└── test-projectState.json -``` - -### Monorepo - -Sometimes, it's helpful to have multiple OpenFn projects all stored in the same -repo, even if they don't use the same workflows (i.e., even if they don't share -a `project.yaml` file.) - -``` -your-git-monorepo -├── project-a -│ ├── config.json -│ ├── projectState.json -│ └── projectSpec.yaml -└── project-b - ├── config.json - ├── projectState.json - └── projectSpec.yaml -``` - -:::tip A sync in time, saves nine - -When you sync changes from OpenFn to GitHub, the `projectSpec.yaml` file in your -repository will be updated with the changes made to the project in OpenFn. For a -project with a directory structure that uses relative paths for job code, OpenFn -will respect the structure when syncing changes for backup. All job code will be -written into their respective files. Newer ones will be kept inline (in the -body) of the `projectSpec.yaml` file. - -When you keep job code in relative file paths, ensure to update the -`projectSpec.yaml` file based on changes to the files or paths in your project -repository. A GitHub action is automatically triggered to push changes to OpenFn -ensuring that future syncs are not affected. Changes can include adding, -renaming, deleting a file or updating a file path. - -::: - ## Troubleshooting ### Github Sync Error: Unexpected inputs provided: ["snapshots"] From 307096b1903e68bb1954b8acec508f947cd3025d Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 17:33:46 +0100 Subject: [PATCH 14/14] typos --- docs/build-for-developers/cli-sync.md | 8 ++++---- docs/deploy/portability.md | 4 ++-- docs/manage-projects/link-to-gh.md | 16 ++++++++-------- 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index 63df02035eb2..ab133a068fe9 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -137,13 +137,13 @@ steps: ``` The `next` key on each step defines any downstream edges for that step - ie, the -steps to execute next. In the example above, the `webbhook` step is executed +steps to execute next. In the example above, the `webhook` step is executed first (determined by the `start` key), and it defines a single edge to `my-step`, which always executes. The actual code for each step lives in its own .js file. You can modify the code freely and sync it back to the server any time. If you want to rename a step, -make sure to to update the file name of the step and the path in the +make sure to update the file name of the step and the path in the `expression` key in `workflow.yaml`. ### openfn.yaml @@ -172,7 +172,7 @@ for help setting up a token. If you connect to multiple OpenFn projects or apps, you can create a `.env` file and set any environment variables you need there. The CLI will load this file -and report which keys its using. Values in your `.env` file will be preferred to +and report which keys it's using. Values in your `.env` file will be preferred to those defined in your system. You can also pass `--api-key` directly as a flag to most commands. @@ -342,7 +342,7 @@ is useful for conflict resolution. ## GitHub You can configure a Project to sync automatically to GitHub. This means that -commit to GitHub will automatically deploy changes to an OpenFn Project; and +commits to GitHub will automatically deploy changes to an OpenFn Project; and pressing Save & Sync in the app will commit back to GitHub. Under the hood, GitHub Sync uses the CLI's `pull` and `deploy` commands, diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index e8e40608c02f..cc6d1474526c 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -47,7 +47,7 @@ myProject/ :::info Directory Structure -There are commonly used 3 directory structure for OpenFn projects namely: +There are 3 commonly used directory structures for OpenFn projects, namely: standard, production & test, and monorepo. To learn more, please see the OpenFn [GitHub configuration documentation](/documentation/link-to-GitHub#structuring-your-github-repository). @@ -408,7 +408,7 @@ OpenFn pull or deploy in your projects. #### Description This error occurs when you run `openfn deploy` and there is a mismatch between -between IDs of workflows in your projectSpec and your OpenFn instance. When this +IDs of workflows in your projectSpec and your OpenFn instance. When this occurs, the error will be written out in an error object as shown below: ``` diff --git a/docs/manage-projects/link-to-gh.md b/docs/manage-projects/link-to-gh.md index 8f36c8b3b0ad..0fd4b84c92e4 100644 --- a/docs/manage-projects/link-to-gh.md +++ b/docs/manage-projects/link-to-gh.md @@ -4,7 +4,7 @@ sidebar_label: GitHub Sync slug: /link-to-GitHub --- -GitHub Sync enables two-way sync between OpenFn project and your GitHub +GitHub Sync enables two-way sync between an OpenFn project and your GitHub repository. This means changes made to a Project in OpenFn can commit back to your GitHub @@ -23,7 +23,7 @@ Users can configure their projects to access one or more repositories on GitHub. Users must have admin access to the GitHub Repo to ensure the OpenFn application is installed. -To configure your project to use Github sync, follow the following steps: +To configure your project to use Github sync, follow these steps: 1. Navigate to `Project Settings > Sync to GitHub` . @@ -50,7 +50,7 @@ To configure your project to use Github sync, follow the following steps: ::: 4. Choose your preferred repository and branch you'd like to connect your - project + project to ![Configure](/img/github-options.webp) @@ -79,7 +79,7 @@ To configure your project to use Github sync, follow the following steps: it to GitHub to start the version control process. If you choose to instead take an existing `project.yaml` file from GitHub and overwrite your current OpenFn project, you won't be able to recover your existing workflows on - OpenFn. This is feature that covers certain advanced use-cases, and unless + OpenFn. This is a feature that covers certain advanced use-cases, and unless you know what you're doing you should start by syncing from "OpenFn to GitHub". @@ -109,7 +109,7 @@ interface. After clicking that link, you can follow the steps below: ![Permissions](/img/lightning_gh_permissions.webp) 4. When you're done making changes on GitHub, head back to OpenFn and refresh - the connection lists with the the 🔄 button next to the drop down list of + the connection lists with the 🔄 button next to the drop down list of available installations. ## Using Version Control & Managing Changes @@ -122,7 +122,7 @@ project. ### Sync from OpenFn to GitHub This sync pushes changes from your OpenFn project to GitHub. This sync operation -will trigger a `openfn pull` action workflow on your connected Github repository +will trigger an `openfn pull` action workflow on your connected Github repository , which will pull the latest configuration from the OpenFn app and save it as code in the `project.yaml` file on your repository. @@ -154,7 +154,7 @@ When you trigger `Save & Sync` in a workflow, your new changes and _previous_ uncommitted changes (if any) to your project's resources (including other workflows) will be committed to GitHub. I.e., if there are other, uncommitted changes by either you or someone else to other workflows in the project, they’ll -also show up in that sync also. +also show up in that sync. ::: @@ -181,7 +181,7 @@ to OpenFn. From v2.7.19, OpenFn deploy and pull actions now support the use of relative paths in project spec. Consequently, projects with directory structure that uses relative paths for job code in project spec, automatically gets packaged and -deployed without the user having to copy changes into the projct spec. This new +deployed without the user having to copy changes into the project spec. This new approach gives developers more flexibility to better manage their job code in individual files rather than having all the code in the `projectSpec.yaml` file.