From 07a2e17c06e5a2e3eb9a9fa91e2916aa4a4391a5 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 30 Apr 2026 12:12:35 +0100 Subject: [PATCH 01/11] update cli intro --- docs/build-for-developers/cli-intro.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/build-for-developers/cli-intro.md b/docs/build-for-developers/cli-intro.md index 944c7712318a..bf58e97c5c47 100644 --- a/docs/build-for-developers/cli-intro.md +++ b/docs/build-for-developers/cli-intro.md @@ -11,10 +11,10 @@ workflows directly from the command line. It’s simple to install, works on macOS, Windows, and Linux, and offers a range of functionality to enhance your developer experience with OpenFn. You can use the OpenFn CLI to: -- Securely run OpenFn steps and workflows +- Sync workflows between OpenFn and a local filesystem or GitHub +- Securely run OpenFn workflows - Troubleshoot and debug OpenFn steps -- Access adaptor documentation -- Deploy workflows to OpenFn +- Read and write Collections data --- @@ -25,7 +25,7 @@ Before you begin with the @openfn/cli, make sure to setup some key tooling: 1. **Code Editor:** Ensure you have a code editor installed on your machine. You can use popular editors like [VS Code](https://code.visualstudio.com/) or [Sublime](https://www.sublimetext.com/). -2. **Node.js:** Install Node.js (version 18 or later). For Linux, Windows, or +2. **Node.js:** Install Node.js (version 24 or later). For Linux, Windows, or macOS, use a version manager like [nvm](https://github.com/nvm-sh/nvm) or [asdf](https://asdf-vm.com/guide/getting-started.html). Or [install Node.js directly](https://kinsta.com/blog/how-to-install-node-js/) @@ -57,7 +57,7 @@ The word `openfn` will invoke the CLI. The word `test` will invoke the test command.
-Expand to see the expected output. +Expand to see the expected output ``` [CLI] ♦ Versions: From 34b321a7cfdc8a4c03303ce4ceaa5f9e9827a633 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 30 Apr 2026 16:56:34 +0100 Subject: [PATCH 02/11] start docs --- docs/build-for-developers/cli-sync.md | 197 ++++++++++++++++++++++++++ sidebars-main.js | 3 +- 2 files changed, 199 insertions(+), 1 deletion(-) create mode 100644 docs/build-for-developers/cli-sync.md diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md new file mode 100644 index 000000000000..5469e1cc5e46 --- /dev/null +++ b/docs/build-for-developers/cli-sync.md @@ -0,0 +1,197 @@ +--- +title: OpenFn Sync +sidebar_label: OpenFn Sync +slug: /sync +--- + +OpefnFn projects are fully portable, which means they can be moved to different +places. + +You can create a project in the app, download it to your local machine to +develop offline, re-upload it to the app, or even deploy it to a different +OpenFn server. + +We call this OpenFn Sync, and it's one of the most powerful features affording +by OpenFn Projects. + +## What is a Project? + +A Project basically set of Workflows (and maybe some configuration) which +solves, automates or integrates some business function. It usually lives in the +app, but it can just be some files on a file system. + +Inside the OpenFn app, a Project is a top level entity backed by a bunch of +database tables. + +The app Project includes a bunch of extra stuff: like configuring Channels, +tracking the run history, saving dataclips, and providing support from the AI +assistant. + +The Project can also exist on the file system, in which case it's a bunch of +files which can be read and executed by the CLI. This local representation of a +project is pretty bare-bones: you'll only find workflows and code here. + +Multiple related projects can exist on the file system at any one time. Each +will ive in a single project file. You can "check out" or "expand" one project +at a time to a local folder, which will create one file for each workflow and +one file for each step. + +Sometimes, We call this superset of all known, related, distributed projects a +Workspace. + +Not all artifacts of a Project are included in a sync. Generally we sync the +project's workflows and some of its options. But we do not sync any associated +data, credential values, usage history or AI sessions. + +## Authorization + +Before you use the CLI to fetch anything from the app, you'll need to provide +some authorization. + +The best way to do this is to set an environment variable called +`OPENFN_API_KEY`. Set it to the value of your +[Personal Access Token](https://docs.openfn.org/documentation/api-tokens#about-api-tokens). + +::: info Personal Access Tokens + +See +[Create and Manage API Tokens](https://docs.openfn.org/documentation/api-tokens) +for help setting up a token. + +::: + +If you connect to multiple OpenFn projects or apps, you can create a `.env` file +and set any environment variables you need there. The CLI will load this file +and report which keys its using. Values in your `.env` file will be preferred to +those defined in your system. + +You can also + +::: info + +This guide assumes you want to sync with our hosted SaaS app at app.openfn.org + +You can sync with a different OpenFn instance by setting the env var +`OPENFN_ENDPOINT` or by passing the `--endpoint` argument to most commands. + +::: + +## Downloading a Project + +To pull a project down from the app to your local machine, run: + +```bash +openfn project pull +``` + +After you've pulled a project for the first time, you don't need to specify the +UUID again (in any commands). + +The `project pull` command does three things: + +- If you do not have an `openfn.yaml` file, it'll create one +- It will _fetch_ your project from the app and save it into a single file at + `.projects/main@app.openfn.org.yaml` +- It will _checkout_ that project onto your file system, expanding workflows and + steps to their own files. + +This will create a file in your working directory called +`.projects/main@app.openfn.org.yaml`. + +## Understanding a Project Structure + +Pulling a project from the app will create a file structure that looks like +this: + +``` +├── openfn.yaml +├── .projects +│ ├── main@app.staging.openfn.org.yaml +└── workflows + ├── my-workflow + │ ├── my-workflow.yaml + │ ├── job.js +``` + +There are three key files to understand here: + +### project.yaml + +The project file saves a copy of the whole state of a project as saved in the +app. If you look inside you'll see the workflows represented as0 plain text. + +A project file is named like `@.yaml`. The alias is a local name +used to refer to a particular version of the project. + +The project file should not be edited locally as changes will be dropped on the +next fetch. + +You can fetch as many projects as you like, and each will be saved to its own +project.yaml file. + +The `.projects` folder can and should be committed to source control. + +### workflows + +Having your whole project inside a single file isn't actually a great way to +read or edit workflows. So the CLI can "checkout" or "expand" a project file +onto the file system. + +Checking out is the process of writing each workflow to a workflow.yaml file and +each step to a step.js file. This all lives in the `workflows` directory. + +Here you can edit files as much as you like, and changes will be tracked when +you push/deploy back to the app. + +You can only check out on project a time. This is actually great for working +with git, because you can checkout two projects on different branches and +compare/merge them against each other directly. + +### openfn.yaml + +This is a top-level configuration file + +We call this file system a Workspace (not a project, because technically it +holds multiple projects) + +At any time, one project can be "checked out" or "expanded". That means the the +contents of the `workflows` folder represents that specific project. You cannot +checkout two versions of the same workflow at once. + +- `openfn.yaml` contains settings and metadata about the checked out project +- `.projects` contains a yaml file for each project. This is a local copy of a + remote project. It should always reflect an app's view of a project. You + should generally not edit these files. +- `workflows` contains all the workflows for the checked out project. These are + compatible with the CLI. + +## Aliases + +Rather than identify a local project with a long-winded UUID or id string, you +can use an _alias_. + +Each local project is saved to a file like `main@app.openfn.org.yaml`, where the +`main` bit is the local alias of a project. + +You can pull a project and set the alias at the same time by running + +```bash +openfn project pull --alias dev +``` + +This will save the project to `dev@app.openfn.org.yaml` + +To change the alias, you can simply rename the file. Whatever goes before the +`@` will be treated as the alias. + +## Running workflows in projects + +## Deploying a Project + +## Sandboxes + +## GitHub + +## Cheatsheet + +An OpenFn project can be represented as a single yaml file diff --git a/sidebars-main.js b/sidebars-main.js index cc0bc31491ca..6cb20f464cbd 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -124,9 +124,10 @@ module.exports = { items: [ 'build-for-developers/cli-intro', 'build-for-developers/cli-usage', + 'build-for-developers/cli-sync', + 'build-for-developers/cli-collections', 'build-for-developers/cli-walkthrough', 'build-for-developers/cli-challenges', - 'build-for-developers/cli-collections', ], }, { From ceefdc4854c93806440d58708047fb7b7ff5fcd1 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 30 Apr 2026 17:21:57 +0100 Subject: [PATCH 03/11] more docs --- docs/build-for-developers/cli-sync.md | 92 ++++++++++++++++++++++++++- 1 file changed, 90 insertions(+), 2 deletions(-) diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index 5469e1cc5e46..dcafce4e74c2 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -65,7 +65,7 @@ and set any environment variables you need there. The CLI will load this file and report which keys its using. Values in your `.env` file will be preferred to those defined in your system. -You can also +You can also pass `--api-key` directly as a flag to most commands. ::: info @@ -184,14 +184,102 @@ This will save the project to `dev@app.openfn.org.yaml` To change the alias, you can simply rename the file. Whatever goes before the `@` will be treated as the alias. +## Checking Out + +You can checkout a project any time with: + +```bash +openfn project checkout +``` + +This will update your local workflows folder with the target project. + +If a checkout will cause changes to be lost (ie, you've changed a step.js file +but haven't deployed it), you will be warned. Add `--force` to ignore the +change, or run `openfn project clean` to wipe and reset the `workflows` folder. + +Checking out will only modify files managed by the CLI - basically workflow and +step files. If you have other files in the file system (like state files or test +files) they will be untouched. + ## Running workflows in projects +You can execute any workflow in the checked out project by name: + +```bash +openfn my-workflow +``` + +The CLI will find the workflow in your `workflows` folder and run it. You can +pass state via `-s` and set log levels as per usual. + +When running a workflow by name like this, you get two benefits: + +- **Credentials** are loaded automatically from the credential map in + `openfn.yaml`, so you don't need to pass `--credential-map` +- **Collections** use the server configured in `openfn.yaml`, so you don't need + to pass `--collections-endpoint` or anything. + ## Deploying a Project +To push your local changes back to the app, run: + +```bash +openfn project deploy +``` + +This will take your currently checked out project and push it to the app. It +will also report what has changed in the local project. + +Before uploading, the CLI fetches the latest version of the project from the app +and checks for **divergence** — meaning it checks whether any of your +locally-changed workflows have also been edited in the app since you last +pulled. If they have, the deploy will fail with an error to prevent you from +accidentally overwriting someone else's work. + +If you want to push anyway, pass `--force`: + +```bash +openfn project deploy --force +``` + +To preview what would change without actually uploading anything, use +`--dry-run`. This will log the final update payload that would be sent to the +app (as a JSON structure). + +You can deploy the checked out project as a new project on the target app by +adding the `--new` command. This is only available if you have superuser +privileges on the target instance. + +You can also deploy the checked out project to another app project by passing +its alias, id or uuid: + +``` +openfn project deploy main +``` + +If you currently have a development sandbox checked out, this would merge it +straight into the main app project. + +Note that you have to have fetched the target project locally before you can +deploy it. + ## Sandboxes ## GitHub ## Cheatsheet -An OpenFn project can be represented as a single yaml file +| Command | Description | +| ---------------------------------------- | ------------------------------------------------------------------------------ | ----- | --------------------------------------- | +| `openfn project pull ` | Pull a project from the app for the first time | +| `openfn project pull` | Re-pull the current project | +| `openfn project pull --alias dev` | Pull and set a local alias | +| `openfn project fetch ` | Switch to a different local project | +| `openfn project deploy` | Deploy checked-out project to the app | +| `openfn project deploy --dry-run` | Try a deploy but skip the upload step | +| `openfn project deploy --force` | Force the checked out project to be uploaded, ignoring any divergence warnings | +| `openfn ` | Run a workflow in the checked-out project | +| `openfn project clean` | Delete the `workflows` folder and all contents, then check out the project | From e5358103a5c87788c26bb020a1adc607eb0ab0c2 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 17:40:47 +0100 Subject: [PATCH 04/11] revision --- docs/build-for-developers/cli-sync.md | 130 ++++++++++++++++++-------- 1 file changed, 89 insertions(+), 41 deletions(-) diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index dcafce4e74c2..87c62e03919d 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -1,6 +1,6 @@ --- title: OpenFn Sync -sidebar_label: OpenFn Sync +sidebar_label: Sync slug: /sync --- @@ -16,28 +16,40 @@ by OpenFn Projects. ## What is a Project? -A Project basically set of Workflows (and maybe some configuration) which -solves, automates or integrates some business function. It usually lives in the -app, but it can just be some files on a file system. +A Project is a set of Workflows which solves, automates or integrates some +business function. -Inside the OpenFn app, a Project is a top level entity backed by a bunch of -database tables. +A Project lives in the OpenFn app (either on the cloud Saas instance or on a +privately deployed instance ), but it also can exist as files on a file system. + +Attached to every Project is some metadata (like a name and a description) and +some configuration like Credentials and Collections. + +Inside the OpenFn app, a Project is a billable top level entity, where all the +workflows and configuration are saved to database tables. The app Project includes a bunch of extra stuff: like configuring Channels, -tracking the run history, saving dataclips, and providing support from the AI +tracking the run history, saving dataclips, and chat sessions with the AI assistant. -The Project can also exist on the file system, in which case it's a bunch of +The Project can also exist on a local file system, in which case it's a bunch of files which can be read and executed by the CLI. This local representation of a project is pretty bare-bones: you'll only find workflows and code here. Multiple related projects can exist on the file system at any one time. Each -will ive in a single project file. You can "check out" or "expand" one project +will live in a single project file. You can "check out" or "expand" one project at a time to a local folder, which will create one file for each workflow and one file for each step. -Sometimes, We call this superset of all known, related, distributed projects a -Workspace. +It's typical that a single conceptual project - that is, the code and +configuration which drives a business function - exists in several places at +once. It might have several representations in the app through sandboxes; be +backed up to GitHub; run locally on a developer's machine; and be distributed to +several remote instances to run in production. + +Sometimes, we call this superset of all known, related, distributed Projects a +Workspace. The Sync problem is the question of how code and configuration is +copied/deployed/replicated across Project instances. Not all artifacts of a Project are included in a sync. Generally we sync the project's workflows and some of its options. But we do not sync any associated @@ -52,7 +64,7 @@ The best way to do this is to set an environment variable called `OPENFN_API_KEY`. Set it to the value of your [Personal Access Token](https://docs.openfn.org/documentation/api-tokens#about-api-tokens). -::: info Personal Access Tokens +:::info Personal Access Tokens See [Create and Manage API Tokens](https://docs.openfn.org/documentation/api-tokens) @@ -67,9 +79,10 @@ those defined in your system. You can also pass `--api-key` directly as a flag to most commands. -::: info +:::info -This guide assumes you want to sync with our hosted SaaS app at app.openfn.org +This guide assumes you want to sync with our hosted SaaS app at +[app.openfn.org](https://app.openfn.org) You can sync with a different OpenFn instance by setting the env var `OPENFN_ENDPOINT` or by passing the `--endpoint` argument to most commands. @@ -84,19 +97,34 @@ To pull a project down from the app to your local machine, run: openfn project pull ``` +This will create a file in your working directory called +`.projects/main@app.openfn.org.yaml`. + +:::info + +Every Project in the app has a unique identifier, called a UUID, which is used +to reference it. It is a 32 digit number of the form +`a6cc5bdd-b04f-4413-b4b8-132a5115acac` + +You can copy a Project's UUID from the URL by browsing to it in the app. It's +the long string after `projects`. + +For example, the UUID is the bold section in: +{'https://openfn.org/projects/'}{'abc087dd-3963-4260-8d09-ced2e1ff2bb0'}{'/w'} + +::: + After you've pulled a project for the first time, you don't need to specify the -UUID again (in any commands). +UUID again. You can use the alias, the id, or leave the identifier blank to use +the currently checked out project. The `project pull` command does three things: - If you do not have an `openfn.yaml` file, it'll create one -- It will _fetch_ your project from the app and save it into a single file at - `.projects/main@app.openfn.org.yaml` -- It will _checkout_ that project onto your file system, expanding workflows and - steps to their own files. - -This will create a file in your working directory called -`.projects/main@app.openfn.org.yaml`. +- It will _fetch_ (download) your project from the app and save it into a single + file at `.projects/main@app.openfn.org.yaml` +- It will _checkout_ (expand) that project onto your file system, expanding + workflows and steps to their own files. ## Understanding a Project Structure @@ -118,13 +146,14 @@ There are three key files to understand here: ### project.yaml The project file saves a copy of the whole state of a project as saved in the -app. If you look inside you'll see the workflows represented as0 plain text. +app. If you look inside you'll see the workflows represented as plain text. A project file is named like `@.yaml`. The alias is a local name -used to refer to a particular version of the project. +used to refer to a particular version of the project. The domain comes from the +OpenFn instance the project was downloaded from. -The project file should not be edited locally as changes will be dropped on the -next fetch. +The project file should not be edited locally as any changes will be dropped on +the next fetch. You can fetch as many projects as you like, and each will be saved to its own project.yaml file. @@ -149,21 +178,10 @@ compare/merge them against each other directly. ### openfn.yaml -This is a top-level configuration file - -We call this file system a Workspace (not a project, because technically it -holds multiple projects) - -At any time, one project can be "checked out" or "expanded". That means the the -contents of the `workflows` folder represents that specific project. You cannot -checkout two versions of the same workflow at once. - -- `openfn.yaml` contains settings and metadata about the checked out project -- `.projects` contains a yaml file for each project. This is a local copy of a - remote project. It should always reflect an app's view of a project. You - should generally not edit these files. -- `workflows` contains all the workflows for the checked out project. These are - compatible with the CLI. +This is a top-level configuration file which can mostly be ignored. It is used +by OpenFn tooling to recognise a project root folder. It also holds +configuration options for all local projects, and metadata about the currently +checked out project. ## Aliases @@ -266,8 +284,38 @@ deploy it. ## Sandboxes +The CLI is fully compatible with sandboxes. Treat them like any other project: +fetch them for the first time with their UUID. + +Use the `checkout` command to switch between sandboxes and projects locally. +Remember you can only have a single project checked out at once. The CLI will +warn you if a checkout will cause you to lose any local changes. + +When fetching a sandbox, by default the project alias will be set to the sandbox +name. + +You can merge two projects locally with `openfn merge`, and deploy the resulting +project to the app (you'll likely have to force push the change). This is useful +for conflict resolution. + ## GitHub +You can configure a Project to sync automatically to GitHub. This means that +commit to GitHub will automatically deploy changes to an OpenFn Project; and +pressing Save & Sync in the app will commit back to GitHub. + +Under the hood, GitHub Sync uses the CLI's `pull` and `deploy` commands, +triggered by GitHub Actions, to sync your projects. + +Note that by default, GitHub Sync will use the legacy format, with `state.json`, +`project.yaml` and `config.json` files. When setting up a new GitHub Sync, you +can choose to select the v2 format. V2 sync is only suitable for pulling a +single Project per branch to github, as multiple projects will overwrite the +same `workflows/` folder. + +See [Version Control](/documentation/link-to-GitHub) for more details about +GitHub Sync. + ## Cheatsheet | Command | Description | From c68ee6921bae5a55e575f316fe58841d1d743aa8 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 17:43:16 +0100 Subject: [PATCH 05/11] claude typos --- docs/build-for-developers/cli-sync.md | 28 +++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index 87c62e03919d..dd147132df67 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -4,14 +4,14 @@ sidebar_label: Sync slug: /sync --- -OpefnFn projects are fully portable, which means they can be moved to different +OpenFn projects are fully portable, which means they can be moved to different places. You can create a project in the app, download it to your local machine to develop offline, re-upload it to the app, or even deploy it to a different OpenFn server. -We call this OpenFn Sync, and it's one of the most powerful features affording +We call this OpenFn Sync, and it's one of the most powerful features afforded by OpenFn Projects. ## What is a Project? @@ -19,8 +19,8 @@ by OpenFn Projects. A Project is a set of Workflows which solves, automates or integrates some business function. -A Project lives in the OpenFn app (either on the cloud Saas instance or on a -privately deployed instance ), but it also can exist as files on a file system. +A Project lives in the OpenFn app (either on the cloud SaaS instance or on a +privately deployed instance), but it also can exist as files on a file system. Attached to every Project is some metadata (like a name and a description) and some configuration like Credentials and Collections. @@ -172,7 +172,7 @@ each step to a step.js file. This all lives in the `workflows` directory. Here you can edit files as much as you like, and changes will be tracked when you push/deploy back to the app. -You can only check out on project a time. This is actually great for working +You can only check out one project at a time. This is actually great for working with git, because you can checkout two projects on different branches and compare/merge them against each other directly. @@ -197,7 +197,7 @@ You can pull a project and set the alias at the same time by running openfn project pull --alias dev ``` -This will save the project to `dev@app.openfn.org.yaml` +This will save the project to `dev@app.openfn.org.yaml`. To change the alias, you can simply rename the file. Whatever goes before the `@` will be treated as the alias. @@ -266,7 +266,7 @@ To preview what would change without actually uploading anything, use app (as a JSON structure). You can deploy the checked out project as a new project on the target app by -adding the `--new` command. This is only available if you have superuser +adding the `--new` flag. This is only available if you have superuser privileges on the target instance. You can also deploy the checked out project to another app project by passing @@ -294,7 +294,7 @@ warn you if a checkout will cause you to lose any local changes. When fetching a sandbox, by default the project alias will be set to the sandbox name. -You can merge two projects locally with `openfn merge`, and deploy the resulting +You can merge two projects locally with `openfn project merge`, and deploy the resulting project to the app (you'll likely have to force push the change). This is useful for conflict resolution. @@ -318,12 +318,12 @@ GitHub Sync. ## Cheatsheet -| Command | Description | -| ---------------------------------------- | ------------------------------------------------------------------------------ | ----- | --------------------------------------- | -| `openfn project pull ` | Pull a project from the app for the first time | -| `openfn project pull` | Re-pull the current project | -| `openfn project pull --alias dev` | Pull and set a local alias | -| `openfn project fetch ` | Pull a project from the app for the first time | +| `openfn project pull` | Re-pull the current project | +| `openfn project pull --alias dev` | Pull and set a local alias | +| `openfn project fetch ` | Fetch a project without checking it out | | `openfn project` | List all local projects in the current working folder | | `openfn project checkout ` | Switch to a different local project | | `openfn project deploy` | Deploy checked-out project to the app | From ce39e8c90de44993e997fbb27cbd7f9c8d7debe6 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 17:59:37 +0100 Subject: [PATCH 06/11] thinking about structure --- docs/build-for-developers/cli-sync.md | 28 +- docs/deploy/portability-v1.md | 439 ++++++++++++++++++++++++++ docs/deploy/portability.md | 61 +++- 3 files changed, 498 insertions(+), 30 deletions(-) create mode 100644 docs/deploy/portability-v1.md diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md index dd147132df67..fbd5399b1ffb 100644 --- a/docs/build-for-developers/cli-sync.md +++ b/docs/build-for-developers/cli-sync.md @@ -11,8 +11,8 @@ You can create a project in the app, download it to your local machine to develop offline, re-upload it to the app, or even deploy it to a different OpenFn server. -We call this OpenFn Sync, and it's one of the most powerful features afforded -by OpenFn Projects. +We call this OpenFn Sync, and it's one of the most powerful features afforded by +OpenFn Projects. ## What is a Project? @@ -126,7 +126,7 @@ The `project pull` command does three things: - It will _checkout_ (expand) that project onto your file system, expanding workflows and steps to their own files. -## Understanding a Project Structure +## Understanding Project Structure Pulling a project from the app will create a file structure that looks like this: @@ -266,8 +266,8 @@ To preview what would change without actually uploading anything, use app (as a JSON structure). You can deploy the checked out project as a new project on the target app by -adding the `--new` flag. This is only available if you have superuser -privileges on the target instance. +adding the `--new` flag. This is only available if you have superuser privileges +on the target instance. You can also deploy the checked out project to another app project by passing its alias, id or uuid: @@ -294,9 +294,9 @@ warn you if a checkout will cause you to lose any local changes. When fetching a sandbox, by default the project alias will be set to the sandbox name. -You can merge two projects locally with `openfn project merge`, and deploy the resulting -project to the app (you'll likely have to force push the change). This is useful -for conflict resolution. +You can merge two projects locally with `openfn project merge`, and deploy the +resulting project to the app (you'll likely have to force push the change). This +is useful for conflict resolution. ## GitHub @@ -318,12 +318,12 @@ GitHub Sync. ## Cheatsheet -| Command | Description | -| -------------------------------------------- | ------------------------------------------------------------------------------ | -| `openfn project pull ` | Pull a project from the app for the first time | -| `openfn project pull` | Re-pull the current project | -| `openfn project pull --alias dev` | Pull and set a local alias | -| `openfn project fetch ` | Fetch a project without checking it out | +| Command | Description | +| ---------------------------------------- | ------------------------------------------------------------------------------ | +| `openfn project pull ` | Pull a project from the app for the first time | +| `openfn project pull` | Re-pull the current project | +| `openfn project pull --alias dev` | Pull and set a local alias | +| `openfn project fetch ` | Fetch a project without checking it out | | `openfn project` | List all local projects in the current working folder | | `openfn project checkout ` | Switch to a different local project | | `openfn project deploy` | Deploy checked-out project to the app | diff --git a/docs/deploy/portability-v1.md b/docs/deploy/portability-v1.md new file mode 100644 index 000000000000..679872d94e2c --- /dev/null +++ b/docs/deploy/portability-v1.md @@ -0,0 +1,439 @@ +--- +title: Portability v1 (legacy) +--- + +The portability specification allows for the representations of entire workflow +projects "as code"; lets user moves between various deployment pathways (such as +cloud, local, hosted); and proposes a globally-applicable way of defining +workflow automation and system integration rules that might be applied across +workflow-engines/integration platforms across the sector. + +Nothing about the spec _must_ be specific to OpenFn or any one of our individual +products. We envision a future in which software built with Lightning, the +OpenFn Integration Toolkit, and entirely new and different integration/workflow +tools can adopt this specification. + +If you're interested in contributing to the specification, reach out to OpenFn +via the [community forum](https://community.openfn.org), write to us, or suggest +changes by submitting a pull request here. + +:::warning + +This is the legacy version of the OpenFn Portability spec. + +For the latest version, see [Portability](portability) + +::: + +## Projects "as code" + +Entire projects (groups of workflows with their associated triggers, edges, +credentials and jobs) can be represented as code. + +This improves the OpenFn developer experience by (a) allowing workflows to be +built and tested locally; (b) enabling project version control and an audit +trail of project changes; and (c) allowing users to port existing projects +between different instances (i.e., deployments) of Lightning. + +### Directory structure + +Many users keep OpenFn projects in git repositories, and this is a common +structure: + +``` +myProject/ +├── workflow-a/ +│ ├── job-1.js +│ ├── job-2.js +│ └── job-3.js +├── workflow-b/ +│ └── job-4.js +├── project.yaml +├── projectState.json +└── config.json +``` + +:::info Directory Structure + +There are commonly used 3 directory structure for OpenFn projects namely: +standard, production & test, and monorepo. To learn more, please see the OpenFn +[GitHub configuration documentation](/documentation/link-to-GitHub#structuring-your-github-repository). + +::: + +### The project **_spec_** + +The project specification (or "spec") is often saved as a `project.yaml` file. +While most of the spec is written inline, many developers prefer to track their +job bodies in separate `.js` files and they then reference them with a relative +path. + +```yaml +name: openhie-project +description: Some sample +credentials: + jane-smith@test.com-HAPI-FHIR: + owner: jane-smith@test.com + name: HAPI FHIR +workflows: + OpenHIE-Workflow: + name: OpenHIE Workflow + jobs: + FHIR-standard-Data-with-change: + name: FHIR-standard-Data-with-change + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: + path: ./jobs/my-fancy-script.js + + Send-to-OpenHIM-to-route-to-SHR: + name: Send-to-OpenHIM-to-route-to-SHR + adaptor: '@openfn/language-http@latest' + enabled: true + credential: jane-smith@test.com-HAPI-FHIR + body: | + fn(state => { + console.log("hello github integration") + return state + }); + + Notify-CHW-upload-successful: + name: Notify-CHW-upload-successful + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: fn(state => state); + + Notify-CHW-upload-failed: + name: Notify-CHW-upload-failed + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: + path: ./jobs/notify-failure.js + + triggers: + webhook: + type: webhook + edges: + webhook->FHIR-standard-Data-with-change: + source_trigger: webhook + target_job: FHIR-standard-Data-with-change + condition: always + FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR: + source_job: FHIR-standard-Data-with-change + target_job: Send-to-OpenHIM-to-route-to-SHR + condition: on_job_success + Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful: + source_job: Send-to-OpenHIM-to-route-to-SHR + target_job: Notify-CHW-upload-successful + condition: on_job_success + Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed: + source_job: Send-to-OpenHIM-to-route-to-SHR + target_job: Notify-CHW-upload-failed + condition: on_job_failure +``` + +In this spec, you can see the different ways of defining a job's body: + +1. Inline body: Used in the `FHIR-standard-Data-with-change` and + `Send-to-OpenHIM-to-route-to-SHR` jobs. The body is directly written in the + YAML file. + +2. External file reference: Used in both `Notify-CHW-upload-successful` and + `Notify-CHW-upload-failed` jobs. The body is stored in separate files, + referenced by the path key. This allows for better organization of complex + job logic. + +When using file paths: + +- Paths are relative to the location of the `project.yaml` file. +- Ensure that the referenced files exist and contain valid job body code. +- This method is particularly useful for complex jobs or when you want to reuse + job bodies across different projects. + +### The project **_state_** + +The project state is a representation of a particular project as _on a specific +Lightning instance_. It is often saved as `projectState.json` and contains UUIDs +for resources on a particular Lightning deployment. + +```json +{ + "id": "8deff39d-8189-4bd7-9dc7-f9f08e7f2c60", + "name": "openhie-project", + "description": null, + "inserted_at": "2023-08-25T08:57:31", + "updated_at": "2023-08-25T08:57:31", + "scheduled_deletion": null, + "requires_mfa": false, + "project_credentials": { + "jane-smith@test.com-HAPI-FHIR": { + "id": "25f48989-d349-4eb8-99c3-923ebba5b116", + "name": "HAPI FHIR", + "owner": "jane-smith@test.com" + } + }, + "workflows": { + "OpenHIE-Workflow": { + "id": "27ae2937-0959-48b8-a597-b1646aae8c14", + "name": "OpenHIE Workflow", + "jobs": { + "Transform-data-to-FHIR-standard": { + "id": "e44f65bb-5038-4e17-8d93-b63cbe95254a", + "delete": true + }, + "Send-to-OpenHIM-to-route-to-SHR": { + "id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "name": "Send-to-OpenHIM-to-route-to-SHR", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "Notify-CHW-upload-successful": { + "id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b", + "name": "Notify-CHW-upload-successful", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "Notify-CHW-upload-failed": { + "id": "be85df30-0abd-4f8e-be17-501f67e18b8d", + "name": "Notify-CHW-upload-failed", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "FHIR-standard-Data": { + "id": "55016dda-42e3-4ee1-8a9c-24e3f23d42f1", + "delete": true + }, + "FHIR-standard-Data-with-change": { + "id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", + "name": "FHIR-standard-Data-with-change", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + } + }, + "triggers": { + "webhook": { + "id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", + "type": "webhook" + } + }, + "edges": { + "webhook->Transform-data-to-FHIR-standard": { + "id": "b2c7407b-0ae9-4ca5-9d6b-ee624976fa54", + "delete": true + }, + "Transform-data-to-FHIR-standard->Send-to-OpenHIM-to-route-to-SHR": { + "id": "d22ed6f4-26a2-4c85-b261-cc110a6851e6", + "delete": true + }, + "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful": { + "id": "26c12f7f-7806-4008-87cd-6747998f95f4", + "condition": "on_job_success", + "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "source_trigger_id": null, + "target_job_id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b" + }, + "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed": { + "id": "0630ac96-4f67-4de7-8c3d-0bf3f89f80d9", + "condition": "on_job_failure", + "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "source_trigger_id": null, + "target_job_id": "be85df30-0abd-4f8e-be17-501f67e18b8d" + }, + "webhook->FHIR-standard-Data": { + "id": "5ce3a8ed-b9eb-464a-a2cd-ba55adc393c2", + "delete": true + }, + "FHIR-standard-Data->Send-to-OpenHIM-to-route-to-SHR": { + "id": "5f459cd9-2882-4a61-a2cc-ec45e58d4837", + "delete": true + }, + "webhook->FHIR-standard-Data-with-change": { + "id": "75e7f7d8-274b-410d-9600-730bbd535229", + "condition": "always", + "source_job_id": null, + "source_trigger_id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", + "target_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe" + }, + "FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR": { + "id": "1e5ba385-2c49-4241-8cd2-042c99a810ec", + "condition": "on_job_success", + "source_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", + "source_trigger_id": null, + "target_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32" + } + } + } + } +} +``` + +## Using the CLI interact with projects + +The project spec and project state can be used for a variety of reasons, e.g. +one could generate the state and spec as backups of the project or one could +generate these files and use them for auditing and record keeping, etc. The +OpenFn [CLI](https://github.com/OpenFn/kit/tree/main/packages/cli) comes with +commands that can be used to pull project configurations down from a running +Lightning server, and to deploy or push updates to existing projects on a +Lightning server. To learn more about automated version control via pull and +deploy, head over to our [Version Control](../manage-projects/link-to-gh.md) +docs. + +:::info Don't have the CLI yet? + +Install it by running `npm install -g @openfn/cli` + +::: + +Before using the CLI, configure it either by passing in environment variables: + +``` +OPENFN_ENDPOINT=https://app.openfn.org +OPENFN_API_KEY=yourSecretApiToken +``` + +Or through a `config.json` file: + +```json +{ + // Required, can be overridden or set with `OPENFN_API_KEY` env var + "apiKey": "***", + + // Optional: can be set using the -p, defaults to project.yaml + "specPath": "project.yaml", + + // Optional: can be set using -s, defaults to .state.json + "statePath": ".state.json", + + // Optional: defaults to OpenFn.org's API, can be overridden or set with + // `OPENFN_ENDPOINT` env var + "endpoint": "https://app.openfn.org" +} +``` + +More details on the CLI can be found +[here](https://github.com/OpenFn/kit/tree/main/packages/cli#basic-usage). + +### `openfn pull` to generate spec & state + +To generate the spec and state files for an existing project, use: + +```sh +openfn pull {YOUR-PROJECT-UUID} -c ./config.json +``` + +This command will save (or overwrite) a project spec and state file based on the +path you've set in your configuration. + +### `openfn deploy` to create new projects + +To deploy a new project to a Lightning instance from a project spec (without a +project state) file use: + +```sh +openfn deploy -c config.json +``` + +### `openfn deploy` to update existing projects + +With a valid project state defined in your `config.json`, the same +`openfn deploy` command will beam up your changes as described by a difference +between your project spec and what's found on the server. + +```sh +openfn deploy -c config.json +Checking https://demo.openfn.org/api/provision/4adf2644-ed4e-4f97-a24c-ab35b3cb1efa for existing project. +Project found. +[CLI] ♦ Changes: + { + workflows: [ + { + jobs: [ + { +- body: "fn(state => {\n console.log(\"ok\")\n return state\n});" ++ body: "fn(state => {\n console.log(\"some changes here!\")\n return state\n});\n" + } + ... + ... + ... + ] + } + ] + } + +? Deploy? yes +[CLI] ♦ Deployed. +``` + +## Getting Help with the cli + +The cli package comes with an inbuilt `help`. Adding `--help` to a command such +as `openfn deploy --help` will result in a help message describing the command +and the options available when using this command. See an example below + +```sh +openfn deploy --help +openfn deploy + +Deploy a project's config to a remote Lightning instance + +Options: + --version Show version number [boolean] + --help Show help [boolean] + -c, --config, --config-path The location of your config file [default: "./.config.json"] + --no-confirm Skip confirmation prompts (e.g. 'Are you sure?') [boolean] + --describe Downloads the project yaml from the specified instance [boolean] + -l, --log Set the log level [string] + --log-json Output all logs as JSON objects [boolean] + -p, --project-path The location of your project.yaml file [string] + -s, --state-path Path to the state file +``` + +## Troubleshooting + +This section covers solutions to some errors you might come across when using +OpenFn pull or deploy in your projects. + +### Extraneous Workflow ID + +#### Description + +This error occurs when you run `openfn deploy` and there is a mismatch between +between IDs of workflows in your projectSpec and your OpenFn instance. When this +occurs, the error will be written out in an error object as shown below: + +``` +[CLI] ✘ Failed to deploy project openfn-data-buffers-prototype: +{ + "errors": { + "workflows": { + "1-ingest-messages": { + "base": [ + "extraneous parameters: workflow_id" + ] + }, + "2-calculate-indicators": { + "base": [ + "extraneous parameters: workflow_id" + ] + } + } + } +``` + +#### Solution + +Run `openfn pull` to update your local instance and keep IDs in sync, +incorporate your changes and run `openfn deploy` again. + +## Other Versions + +- [Portability Spec v2](portability-versions#v2) +- [Portability Spec v1](portability-versions#v1) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 10f9d2ac8973..1c48ecfb54d0 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -2,30 +2,58 @@ title: Portability --- -## Intent - The portability specification allows for the representations of entire workflow -projects "as code", lets user move between various deployment pathways (cloud, -local, DIY, etc.) and proposes a globally-applicable way of **_specifying -workflow automation_** and **_systems integration_** that might be applied -across workflow-engines/integration platforms across the sector. Nothing about -the spec _must_ be specific to OpenFn or any one of our individual products. We -envision a future in which software built with Lightning, the OpenFn Integration -Toolkit, and entirely new and different integration/workflow tools can adopt -this specification. +projects "as code"; lets user moves between various deployment pathways (such as +cloud, local, hosted); and proposes a globally-applicable way of defining +workflow automation and system integration rules that might be applied across +workflow-engines/integration platforms across the sector. + +Nothing about the spec _must_ be specific to OpenFn or any one of our individual +products. We envision a future in which software built with Lightning, the +OpenFn Integration Toolkit, and entirely new and different integration/workflow +tools can adopt this specification. If you're interested in contributing to the specification, reach out to OpenFn via the [community forum](https://community.openfn.org), write to us, or suggest changes by submitting a pull request here. +:::info + +This document describes v2 of the OpenFn deployment spec, in-line with +[OpenFn Sync](/documentation/sync). + +For the v1 spec (using state, project and config files), see +[Legacy Portability](./portability-v1). + +::: + ## Projects "as code" Entire projects (groups of workflows with their associated triggers, edges, -credentials and jobs) can be represented as code. This improves the OpenFn -developer experience by (a) allowing workflows to be built and tested locally; -(b) enabling project version control and an audit trail of project changes; and -(c) allowing users to port existing projects between different instances (i.e., -deployments) of Lightning. +credentials and jobs) can be represented as code. + +This improves the OpenFn developer experience by: + +1. Allowing workflows to be built and tested locally +2. Enabling project version control and an audit trail of project changes +3. Allowing users to port existing projects between different instances (i.e., + deployments) of Lightning. + +# Joe's Thoughts + +What should go in this file? + +The Sync document is the best equivalent. Should I just reference that? + +This document talks about a portability "spec". But I don't really see that +explained anywhere. Should I rewrite that stuff into a higher level manifesto? + +There's also already a v1 and v2 spec. So actually what I'm doing now is a v3? + +I should chat with Taylor about what to do here. + +I am super unconfortable having multiple versions of the sync workflow on the +site ### Directory structure @@ -395,7 +423,7 @@ OpenFn pull or deploy in your projects. ### Extraneous Workflow ID -#### Description +#### Description This error occurs when you run `openfn deploy` and there is a mismatch between between IDs of workflows in your projectSpec and your OpenFn instance. When this @@ -419,6 +447,7 @@ occurs, the error will be written out in an error object as shown below: } } ``` + #### Solution Run `openfn pull` to update your local instance and keep IDs in sync, From 003cb63343e0f475f3336bbbdd54a2eca5f3c6f0 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 4 May 2026 18:11:22 +0100 Subject: [PATCH 07/11] fiddling with sync --- docs/manage-projects/link-to-gh.md | 38 +++++++++++++++--------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/manage-projects/link-to-gh.md b/docs/manage-projects/link-to-gh.md index 92a8204f03ec..f919be00199b 100644 --- a/docs/manage-projects/link-to-gh.md +++ b/docs/manage-projects/link-to-gh.md @@ -1,31 +1,31 @@ --- -title: Version Control -sidebar_label: Version Control (GitHub Sync) +title: GitHub Sync +sidebar_label: GitHub Sync slug: /link-to-GitHub --- -The version control (GitHub Sync) feature lets users track and manage changes to -their OpenFn projects in GitHub. GitHub Sync enables a 2-way sync between your -OpenFn project and your GitHub repository. By 2-way sync, we mean that you can -**sync changes made to your project on OpenFn to GitHub**, and you can **deploy -changes you've made to your project on GitHub to OpenFn**. +GitHub Sync enables two-way sync between OpenFn project and your GitHub +repository. + +This means changes made to a Project in OpenFn can commit back to your GitHub +repo; and commits pushed to GitHub can update the Project in your OpenFn App. :::info For Cloud Hosted OpenFn Users GitHub Sync is only available in projects that are subscribed to Core, Growth, -Scale or Custom plans on Cloud Hosted OpenFn. Users can however connect their -OpenFn account to GitHub by authenticating OpenFn to access their GitHub account -by navigating to `User Profile` page and clicking ` Connect your GitHub -Account". +Scale or Custom plans on Cloud Hosted OpenFn. + +Users can however connect their OpenFn account to GitHub by authenticating +OpenFn to access their GitHub account by navigating to `User Profile` page and +clicking ` Connect your GitHub Account". ::: -### Configuring your project to use GitHub Sync +### Configuring your Project to use GitHub Sync -Users are able to configure their projects to have access to one or more -repositories on GitHub. To enable sync, the OpenFn project requires a repository -where a GitHub OpenFn application is installed and users are required to have -administrative access to the repositiory. +Users can configure their projects to access one or more repositories on GitHub. +Users must have admin access to the GitHub Repo to ensure the OpenFn application +is installed. To configure your project to use Github sync, follow the following steps: @@ -285,8 +285,6 @@ your-git-monorepo :::tip A sync in time, saves nine -#### Syncing Changes from OpenFn to GitHub - When you sync changes from OpenFn to GitHub, the `projectSpec.yaml` file in your repository will be updated with the changes made to the project in OpenFn. For a project with a directory structure that uses relative paths for job code, OpenFn @@ -298,7 +296,9 @@ When you keep job code in relative file paths, ensure to update the `projectSpec.yaml` file based on changes to the files or paths in your project repository. A GitHub action is automatically triggered to push changes to OpenFn ensuring that future syncs are not affected. Changes can include adding, -renaming, deleting a file or updating a file path. ::: +renaming, deleting a file or updating a file path. + +::: ## Troubleshooting From d57d90f6e89ba949ae89de7e1be28ce14ba934b0 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 12:13:33 +0100 Subject: [PATCH 08/11] remove notes --- docs/deploy/portability.md | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 1c48ecfb54d0..4a89c08988b3 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -39,22 +39,6 @@ This improves the OpenFn developer experience by: 3. Allowing users to port existing projects between different instances (i.e., deployments) of Lightning. -# Joe's Thoughts - -What should go in this file? - -The Sync document is the best equivalent. Should I just reference that? - -This document talks about a portability "spec". But I don't really see that -explained anywhere. Should I rewrite that stuff into a higher level manifesto? - -There's also already a v1 and v2 spec. So actually what I'm doing now is a v3? - -I should chat with Taylor about what to do here. - -I am super unconfortable having multiple versions of the sync workflow on the -site - ### Directory structure Many users keep OpenFn projects in git repositories, and this is a common From 2be5f7f0e91a74d51cc785072d18b38f1f4c0eb0 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:24:54 +0100 Subject: [PATCH 09/11] typo --- docs/deploy/portability.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 4a89c08988b3..54aea57f5eac 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -3,7 +3,7 @@ title: Portability --- The portability specification allows for the representations of entire workflow -projects "as code"; lets user moves between various deployment pathways (such as +projects "as code"; lets users move between various deployment pathways (such as cloud, local, hosted); and proposes a globally-applicable way of defining workflow automation and system integration rules that might be applied across workflow-engines/integration platforms across the sector. From 76822c6447c60c72a20799393e7c5fff51cf38b2 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:26:19 +0100 Subject: [PATCH 10/11] rollback changes --- docs/build-for-developers/cli-intro.md | 10 +++---- docs/manage-projects/link-to-gh.md | 38 +++++++++++++------------- 2 files changed, 24 insertions(+), 24 deletions(-) diff --git a/docs/build-for-developers/cli-intro.md b/docs/build-for-developers/cli-intro.md index bf58e97c5c47..944c7712318a 100644 --- a/docs/build-for-developers/cli-intro.md +++ b/docs/build-for-developers/cli-intro.md @@ -11,10 +11,10 @@ workflows directly from the command line. It’s simple to install, works on macOS, Windows, and Linux, and offers a range of functionality to enhance your developer experience with OpenFn. You can use the OpenFn CLI to: -- Sync workflows between OpenFn and a local filesystem or GitHub -- Securely run OpenFn workflows +- Securely run OpenFn steps and workflows - Troubleshoot and debug OpenFn steps -- Read and write Collections data +- Access adaptor documentation +- Deploy workflows to OpenFn --- @@ -25,7 +25,7 @@ Before you begin with the @openfn/cli, make sure to setup some key tooling: 1. **Code Editor:** Ensure you have a code editor installed on your machine. You can use popular editors like [VS Code](https://code.visualstudio.com/) or [Sublime](https://www.sublimetext.com/). -2. **Node.js:** Install Node.js (version 24 or later). For Linux, Windows, or +2. **Node.js:** Install Node.js (version 18 or later). For Linux, Windows, or macOS, use a version manager like [nvm](https://github.com/nvm-sh/nvm) or [asdf](https://asdf-vm.com/guide/getting-started.html). Or [install Node.js directly](https://kinsta.com/blog/how-to-install-node-js/) @@ -57,7 +57,7 @@ The word `openfn` will invoke the CLI. The word `test` will invoke the test command.
-Expand to see the expected output +Expand to see the expected output. ``` [CLI] ♦ Versions: diff --git a/docs/manage-projects/link-to-gh.md b/docs/manage-projects/link-to-gh.md index f919be00199b..92a8204f03ec 100644 --- a/docs/manage-projects/link-to-gh.md +++ b/docs/manage-projects/link-to-gh.md @@ -1,31 +1,31 @@ --- -title: GitHub Sync -sidebar_label: GitHub Sync +title: Version Control +sidebar_label: Version Control (GitHub Sync) slug: /link-to-GitHub --- -GitHub Sync enables two-way sync between OpenFn project and your GitHub -repository. - -This means changes made to a Project in OpenFn can commit back to your GitHub -repo; and commits pushed to GitHub can update the Project in your OpenFn App. +The version control (GitHub Sync) feature lets users track and manage changes to +their OpenFn projects in GitHub. GitHub Sync enables a 2-way sync between your +OpenFn project and your GitHub repository. By 2-way sync, we mean that you can +**sync changes made to your project on OpenFn to GitHub**, and you can **deploy +changes you've made to your project on GitHub to OpenFn**. :::info For Cloud Hosted OpenFn Users GitHub Sync is only available in projects that are subscribed to Core, Growth, -Scale or Custom plans on Cloud Hosted OpenFn. - -Users can however connect their OpenFn account to GitHub by authenticating -OpenFn to access their GitHub account by navigating to `User Profile` page and -clicking ` Connect your GitHub Account". +Scale or Custom plans on Cloud Hosted OpenFn. Users can however connect their +OpenFn account to GitHub by authenticating OpenFn to access their GitHub account +by navigating to `User Profile` page and clicking ` Connect your GitHub +Account". ::: -### Configuring your Project to use GitHub Sync +### Configuring your project to use GitHub Sync -Users can configure their projects to access one or more repositories on GitHub. -Users must have admin access to the GitHub Repo to ensure the OpenFn application -is installed. +Users are able to configure their projects to have access to one or more +repositories on GitHub. To enable sync, the OpenFn project requires a repository +where a GitHub OpenFn application is installed and users are required to have +administrative access to the repositiory. To configure your project to use Github sync, follow the following steps: @@ -285,6 +285,8 @@ your-git-monorepo :::tip A sync in time, saves nine +#### Syncing Changes from OpenFn to GitHub + When you sync changes from OpenFn to GitHub, the `projectSpec.yaml` file in your repository will be updated with the changes made to the project in OpenFn. For a project with a directory structure that uses relative paths for job code, OpenFn @@ -296,9 +298,7 @@ When you keep job code in relative file paths, ensure to update the `projectSpec.yaml` file based on changes to the files or paths in your project repository. A GitHub action is automatically triggered to push changes to OpenFn ensuring that future syncs are not affected. Changes can include adding, -renaming, deleting a file or updating a file path. - -::: +renaming, deleting a file or updating a file path. ::: ## Troubleshooting From d0b39a8176fedccd55a5664ac3bfa2de87eee6ea Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Tue, 5 May 2026 14:32:47 +0100 Subject: [PATCH 11/11] rollback again --- docs/build-for-developers/cli-sync.md | 333 -------------------------- sidebars-main.js | 1 - 2 files changed, 334 deletions(-) delete mode 100644 docs/build-for-developers/cli-sync.md diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md deleted file mode 100644 index fbd5399b1ffb..000000000000 --- a/docs/build-for-developers/cli-sync.md +++ /dev/null @@ -1,333 +0,0 @@ ---- -title: OpenFn Sync -sidebar_label: Sync -slug: /sync ---- - -OpenFn projects are fully portable, which means they can be moved to different -places. - -You can create a project in the app, download it to your local machine to -develop offline, re-upload it to the app, or even deploy it to a different -OpenFn server. - -We call this OpenFn Sync, and it's one of the most powerful features afforded by -OpenFn Projects. - -## What is a Project? - -A Project is a set of Workflows which solves, automates or integrates some -business function. - -A Project lives in the OpenFn app (either on the cloud SaaS instance or on a -privately deployed instance), but it also can exist as files on a file system. - -Attached to every Project is some metadata (like a name and a description) and -some configuration like Credentials and Collections. - -Inside the OpenFn app, a Project is a billable top level entity, where all the -workflows and configuration are saved to database tables. - -The app Project includes a bunch of extra stuff: like configuring Channels, -tracking the run history, saving dataclips, and chat sessions with the AI -assistant. - -The Project can also exist on a local file system, in which case it's a bunch of -files which can be read and executed by the CLI. This local representation of a -project is pretty bare-bones: you'll only find workflows and code here. - -Multiple related projects can exist on the file system at any one time. Each -will live in a single project file. You can "check out" or "expand" one project -at a time to a local folder, which will create one file for each workflow and -one file for each step. - -It's typical that a single conceptual project - that is, the code and -configuration which drives a business function - exists in several places at -once. It might have several representations in the app through sandboxes; be -backed up to GitHub; run locally on a developer's machine; and be distributed to -several remote instances to run in production. - -Sometimes, we call this superset of all known, related, distributed Projects a -Workspace. The Sync problem is the question of how code and configuration is -copied/deployed/replicated across Project instances. - -Not all artifacts of a Project are included in a sync. Generally we sync the -project's workflows and some of its options. But we do not sync any associated -data, credential values, usage history or AI sessions. - -## Authorization - -Before you use the CLI to fetch anything from the app, you'll need to provide -some authorization. - -The best way to do this is to set an environment variable called -`OPENFN_API_KEY`. Set it to the value of your -[Personal Access Token](https://docs.openfn.org/documentation/api-tokens#about-api-tokens). - -:::info Personal Access Tokens - -See -[Create and Manage API Tokens](https://docs.openfn.org/documentation/api-tokens) -for help setting up a token. - -::: - -If you connect to multiple OpenFn projects or apps, you can create a `.env` file -and set any environment variables you need there. The CLI will load this file -and report which keys its using. Values in your `.env` file will be preferred to -those defined in your system. - -You can also pass `--api-key` directly as a flag to most commands. - -:::info - -This guide assumes you want to sync with our hosted SaaS app at -[app.openfn.org](https://app.openfn.org) - -You can sync with a different OpenFn instance by setting the env var -`OPENFN_ENDPOINT` or by passing the `--endpoint` argument to most commands. - -::: - -## Downloading a Project - -To pull a project down from the app to your local machine, run: - -```bash -openfn project pull -``` - -This will create a file in your working directory called -`.projects/main@app.openfn.org.yaml`. - -:::info - -Every Project in the app has a unique identifier, called a UUID, which is used -to reference it. It is a 32 digit number of the form -`a6cc5bdd-b04f-4413-b4b8-132a5115acac` - -You can copy a Project's UUID from the URL by browsing to it in the app. It's -the long string after `projects`. - -For example, the UUID is the bold section in: -{'https://openfn.org/projects/'}{'abc087dd-3963-4260-8d09-ced2e1ff2bb0'}{'/w'} - -::: - -After you've pulled a project for the first time, you don't need to specify the -UUID again. You can use the alias, the id, or leave the identifier blank to use -the currently checked out project. - -The `project pull` command does three things: - -- If you do not have an `openfn.yaml` file, it'll create one -- It will _fetch_ (download) your project from the app and save it into a single - file at `.projects/main@app.openfn.org.yaml` -- It will _checkout_ (expand) that project onto your file system, expanding - workflows and steps to their own files. - -## Understanding Project Structure - -Pulling a project from the app will create a file structure that looks like -this: - -``` -├── openfn.yaml -├── .projects -│ ├── main@app.staging.openfn.org.yaml -└── workflows - ├── my-workflow - │ ├── my-workflow.yaml - │ ├── job.js -``` - -There are three key files to understand here: - -### project.yaml - -The project file saves a copy of the whole state of a project as saved in the -app. If you look inside you'll see the workflows represented as plain text. - -A project file is named like `@.yaml`. The alias is a local name -used to refer to a particular version of the project. The domain comes from the -OpenFn instance the project was downloaded from. - -The project file should not be edited locally as any changes will be dropped on -the next fetch. - -You can fetch as many projects as you like, and each will be saved to its own -project.yaml file. - -The `.projects` folder can and should be committed to source control. - -### workflows - -Having your whole project inside a single file isn't actually a great way to -read or edit workflows. So the CLI can "checkout" or "expand" a project file -onto the file system. - -Checking out is the process of writing each workflow to a workflow.yaml file and -each step to a step.js file. This all lives in the `workflows` directory. - -Here you can edit files as much as you like, and changes will be tracked when -you push/deploy back to the app. - -You can only check out one project at a time. This is actually great for working -with git, because you can checkout two projects on different branches and -compare/merge them against each other directly. - -### openfn.yaml - -This is a top-level configuration file which can mostly be ignored. It is used -by OpenFn tooling to recognise a project root folder. It also holds -configuration options for all local projects, and metadata about the currently -checked out project. - -## Aliases - -Rather than identify a local project with a long-winded UUID or id string, you -can use an _alias_. - -Each local project is saved to a file like `main@app.openfn.org.yaml`, where the -`main` bit is the local alias of a project. - -You can pull a project and set the alias at the same time by running - -```bash -openfn project pull --alias dev -``` - -This will save the project to `dev@app.openfn.org.yaml`. - -To change the alias, you can simply rename the file. Whatever goes before the -`@` will be treated as the alias. - -## Checking Out - -You can checkout a project any time with: - -```bash -openfn project checkout -``` - -This will update your local workflows folder with the target project. - -If a checkout will cause changes to be lost (ie, you've changed a step.js file -but haven't deployed it), you will be warned. Add `--force` to ignore the -change, or run `openfn project clean` to wipe and reset the `workflows` folder. - -Checking out will only modify files managed by the CLI - basically workflow and -step files. If you have other files in the file system (like state files or test -files) they will be untouched. - -## Running workflows in projects - -You can execute any workflow in the checked out project by name: - -```bash -openfn my-workflow -``` - -The CLI will find the workflow in your `workflows` folder and run it. You can -pass state via `-s` and set log levels as per usual. - -When running a workflow by name like this, you get two benefits: - -- **Credentials** are loaded automatically from the credential map in - `openfn.yaml`, so you don't need to pass `--credential-map` -- **Collections** use the server configured in `openfn.yaml`, so you don't need - to pass `--collections-endpoint` or anything. - -## Deploying a Project - -To push your local changes back to the app, run: - -```bash -openfn project deploy -``` - -This will take your currently checked out project and push it to the app. It -will also report what has changed in the local project. - -Before uploading, the CLI fetches the latest version of the project from the app -and checks for **divergence** — meaning it checks whether any of your -locally-changed workflows have also been edited in the app since you last -pulled. If they have, the deploy will fail with an error to prevent you from -accidentally overwriting someone else's work. - -If you want to push anyway, pass `--force`: - -```bash -openfn project deploy --force -``` - -To preview what would change without actually uploading anything, use -`--dry-run`. This will log the final update payload that would be sent to the -app (as a JSON structure). - -You can deploy the checked out project as a new project on the target app by -adding the `--new` flag. This is only available if you have superuser privileges -on the target instance. - -You can also deploy the checked out project to another app project by passing -its alias, id or uuid: - -``` -openfn project deploy main -``` - -If you currently have a development sandbox checked out, this would merge it -straight into the main app project. - -Note that you have to have fetched the target project locally before you can -deploy it. - -## Sandboxes - -The CLI is fully compatible with sandboxes. Treat them like any other project: -fetch them for the first time with their UUID. - -Use the `checkout` command to switch between sandboxes and projects locally. -Remember you can only have a single project checked out at once. The CLI will -warn you if a checkout will cause you to lose any local changes. - -When fetching a sandbox, by default the project alias will be set to the sandbox -name. - -You can merge two projects locally with `openfn project merge`, and deploy the -resulting project to the app (you'll likely have to force push the change). This -is useful for conflict resolution. - -## GitHub - -You can configure a Project to sync automatically to GitHub. This means that -commit to GitHub will automatically deploy changes to an OpenFn Project; and -pressing Save & Sync in the app will commit back to GitHub. - -Under the hood, GitHub Sync uses the CLI's `pull` and `deploy` commands, -triggered by GitHub Actions, to sync your projects. - -Note that by default, GitHub Sync will use the legacy format, with `state.json`, -`project.yaml` and `config.json` files. When setting up a new GitHub Sync, you -can choose to select the v2 format. V2 sync is only suitable for pulling a -single Project per branch to github, as multiple projects will overwrite the -same `workflows/` folder. - -See [Version Control](/documentation/link-to-GitHub) for more details about -GitHub Sync. - -## Cheatsheet - -| Command | Description | -| ---------------------------------------- | ------------------------------------------------------------------------------ | -| `openfn project pull ` | Pull a project from the app for the first time | -| `openfn project pull` | Re-pull the current project | -| `openfn project pull --alias dev` | Pull and set a local alias | -| `openfn project fetch ` | Fetch a project without checking it out | -| `openfn project` | List all local projects in the current working folder | -| `openfn project checkout ` | Switch to a different local project | -| `openfn project deploy` | Deploy checked-out project to the app | -| `openfn project deploy --dry-run` | Try a deploy but skip the upload step | -| `openfn project deploy --force` | Force the checked out project to be uploaded, ignoring any divergence warnings | -| `openfn ` | Run a workflow in the checked-out project | -| `openfn project clean` | Delete the `workflows` folder and all contents, then check out the project | diff --git a/sidebars-main.js b/sidebars-main.js index 6cb20f464cbd..404fc48fe25d 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -124,7 +124,6 @@ module.exports = { items: [ 'build-for-developers/cli-intro', 'build-for-developers/cli-usage', - 'build-for-developers/cli-sync', 'build-for-developers/cli-collections', 'build-for-developers/cli-walkthrough', 'build-for-developers/cli-challenges',