diff --git a/docs/build-for-developers/cli-intro.md b/docs/build-for-developers/cli-intro.md index 944c7712318a..bf58e97c5c47 100644 --- a/docs/build-for-developers/cli-intro.md +++ b/docs/build-for-developers/cli-intro.md @@ -11,10 +11,10 @@ workflows directly from the command line. It’s simple to install, works on macOS, Windows, and Linux, and offers a range of functionality to enhance your developer experience with OpenFn. You can use the OpenFn CLI to: -- Securely run OpenFn steps and workflows +- Sync workflows between OpenFn and a local filesystem or GitHub +- Securely run OpenFn workflows - Troubleshoot and debug OpenFn steps -- Access adaptor documentation -- Deploy workflows to OpenFn +- Read and write Collections data --- @@ -25,7 +25,7 @@ Before you begin with the @openfn/cli, make sure to setup some key tooling: 1. **Code Editor:** Ensure you have a code editor installed on your machine. You can use popular editors like [VS Code](https://code.visualstudio.com/) or [Sublime](https://www.sublimetext.com/). -2. **Node.js:** Install Node.js (version 18 or later). For Linux, Windows, or +2. **Node.js:** Install Node.js (version 24 or later). For Linux, Windows, or macOS, use a version manager like [nvm](https://github.com/nvm-sh/nvm) or [asdf](https://asdf-vm.com/guide/getting-started.html). Or [install Node.js directly](https://kinsta.com/blog/how-to-install-node-js/) @@ -57,7 +57,7 @@ The word `openfn` will invoke the CLI. The word `test` will invoke the test command.
-Expand to see the expected output. +Expand to see the expected output ``` [CLI] ♦ Versions: diff --git a/docs/build-for-developers/cli-sync.md b/docs/build-for-developers/cli-sync.md new file mode 100644 index 000000000000..ab133a068fe9 --- /dev/null +++ b/docs/build-for-developers/cli-sync.md @@ -0,0 +1,374 @@ +--- +title: OpenFn Sync +sidebar_label: Sync +slug: /sync +--- + +OpenFn projects are fully portable, which means they can be moved to different +places. + +You can create a project in the app, download it to your local machine to +develop offline, re-upload it to the app, or even deploy it to a different +OpenFn server. + +We call this OpenFn Sync, and it's one of the most powerful features afforded by +OpenFn Projects. + +## What is a Project? + +A Project is a set of Workflows which solves, automates or integrates some +business function. + +A Project lives in the OpenFn app (either on the cloud SaaS instance or on a +privately deployed instance), but it also can exist as files on a file system. + +Attached to every Project is some metadata (like a name and a description) and +some configuration like Credentials and Collections. + +Inside the OpenFn app, a Project is a billable top level entity, where all the +workflows and configuration are saved to database tables. + +The app Project includes a bunch of extra stuff: like configuring Channels, +tracking the run history, saving dataclips, and chat sessions with the AI +assistant. + +The Project can also exist on a local file system, in which case it's a bunch of +files which can be read and executed by the CLI. This local representation of a +project is pretty bare-bones: you'll only find workflows and code here. + +Multiple related projects can exist on the file system at any one time. Each +will live in a single project file. You can "check out" or "expand" one project +at a time to a local folder, which will create one file for each workflow and +one file for each step. + +It's typical that a single conceptual project - that is, the code and +configuration which drives a business function - exists in several places at +once. It might have several representations in the app through sandboxes; be +backed up to GitHub; run locally on a developer's machine; and be distributed to +several remote instances to run in production. + +Sometimes, we call this superset of all known, related, distributed Projects a +Workspace. The Sync problem is the question of how code and configuration is +copied/deployed/replicated across Project instances. + +Not all artifacts of a Project are included in a sync. Generally we sync the +project's workflows and some of its options. But we do not sync any associated +data, credential values, usage history or AI sessions. + +## Project Structure + +OpenFn Sync writes a Project to the file system using a number of conventions. +Whether using the CLI or GitHub Sync, a Project has the following structure + +``` +├── openfn.yaml +├── .projects +│ ├── main@app.openfn.org.yaml +└── workflows + ├── my-workflow + │ ├── my-workflow.yaml + │ ├── my-step.js +``` + +Briefly, these files are: + +- `openfn.yaml` declares this folder to be an OpenFn Project, and contains + metadata and settings +- The `projects` folder contains a complete YAML representation of each Project +- The `workflows` folder shows the contents of the project - the steps and edges + and so on. + +Let's look at this structure in a bit more detail. + +### project.yaml + +The project file saves a copy of the whole state of a project as saved in the +app. If you look inside you'll see the workflows represented as plain text. + +A project file is named like `@.yaml`. The alias is a local name +used to refer to a particular version of the project. The domain comes from the +OpenFn instance the project was downloaded from. + +The project file should not be edited locally as any changes will be dropped on +the next fetch. + +You can fetch as many projects as you like, and each will be saved to its own +project.yaml file. + +The `.projects` folder can and should be committed to source control. + +### workflows + +Having your whole project inside a single file isn't actually a great way to +read or edit workflows. So the CLI can "checkout" or "expand" a project file +onto the file system. + +Checking out is the process of writing each workflow to a workflow.yaml file and +each step to a step.js file. This all lives in the `workflows` directory. + +Here you can edit files as much as you like, and changes will be tracked when +you push/deploy back to the app. + +You can only check out one project at a time. This is actually great for working +with git, because you can checkout two projects on different branches and +compare/merge them against each other directly. + +### workflow.yaml + +A workflow.yaml file defines the steps of a workflow and the edges which connect +them. + +``` +id: my-workflow +name: My Workflow +start: webhook +steps: + - id: my-step + name: My Step + adaptor: '@openfn/language-http@7.2.9' + expression: ./my-step.js + - id: webhook + type: webhook + enabled: true + next: + my-step: + disabled: false + condition: always +``` + +The `next` key on each step defines any downstream edges for that step - ie, the +steps to execute next. In the example above, the `webhook` step is executed +first (determined by the `start` key), and it defines a single edge to +`my-step`, which always executes. + +The actual code for each step lives in its own .js file. You can modify the code +freely and sync it back to the server any time. If you want to rename a step, +make sure to update the file name of the step and the path in the +`expression` key in `workflow.yaml`. + +### openfn.yaml + +This is a top-level configuration file which can mostly be ignored. It is used +by OpenFn tooling to recognise a project root folder. It also holds +configuration options for all local projects, and metadata about the currently +checked out project. + +## Authorization + +Before you use the CLI to fetch anything from the app, you'll need to provide +some authorization. + +The best way to do this is to set an environment variable called +`OPENFN_API_KEY`. Set it to the value of your +[Personal Access Token](https://docs.openfn.org/documentation/api-tokens#about-api-tokens). + +:::info Personal Access Tokens + +See +[Create and Manage API Tokens](https://docs.openfn.org/documentation/api-tokens) +for help setting up a token. + +::: + +If you connect to multiple OpenFn projects or apps, you can create a `.env` file +and set any environment variables you need there. The CLI will load this file +and report which keys it's using. Values in your `.env` file will be preferred to +those defined in your system. + +You can also pass `--api-key` directly as a flag to most commands. + +:::info + +This guide assumes you want to sync with our hosted SaaS app at +[app.openfn.org](https://app.openfn.org) + +You can sync with a different OpenFn instance by setting the env var +`OPENFN_ENDPOINT` or by passing the `--endpoint` argument to most commands. + +::: + +## Downloading a Project + +To pull a project down from the app to your local machine, run: + +```bash +openfn project pull +``` + +This will create a file in your working directory called +`.projects/main@app.openfn.org.yaml`. + +:::info + +Every Project in the app has a unique identifier, called a UUID, which is used +to reference it. It is a 32 digit number of the form +`a6cc5bdd-b04f-4413-b4b8-132a5115acac` + +You can copy a Project's UUID from the URL by browsing to it in the app. It's +the long string after `projects`. + +For example, the UUID is the bold section in: +{'https://openfn.org/projects/'}{'abc087dd-3963-4260-8d09-ced2e1ff2bb0'}{'/w'} + +::: + +After you've pulled a project for the first time, you don't need to specify the +UUID again. You can use the alias, the id, or leave the identifier blank to use +the currently checked out project. + +The `project pull` command does three things: + +- If you do not have an `openfn.yaml` file, it'll create one +- It will _fetch_ (download) your project from the app and save it into a single + file at `.projects/main@app.openfn.org.yaml` +- It will _checkout_ (expand) that project onto your file system, expanding + workflows and steps to their own files. + +## Aliases + +Rather than identify a local project with a long-winded UUID or id string, you +can use an _alias_. + +Each local project is saved to a file like `main@app.openfn.org.yaml`, where the +`main` bit is the local alias of a project. + +You can pull a project and set the alias at the same time by running + +```bash +openfn project pull --alias dev +``` + +This will save the project to `dev@app.openfn.org.yaml`. + +To change the alias, you can simply rename the file. Whatever goes before the +`@` will be treated as the alias. + +## Checking Out + +You can checkout a project any time with: + +```bash +openfn project checkout +``` + +This will update your local workflows folder with the target project. + +If a checkout will cause changes to be lost (ie, you've changed a step.js file +but haven't deployed it), you will be warned. Add `--force` to ignore the +change, or run `openfn project clean` to wipe and reset the `workflows` folder. + +Checking out will only modify files managed by the CLI - basically workflow and +step files. If you have other files in the file system (like state files or test +files) they will be untouched. + +## Running workflows in projects + +You can execute any workflow in the checked out project by name: + +```bash +openfn my-workflow +``` + +The CLI will find the workflow in your `workflows` folder and run it. You can +pass state via `-s` and set log levels as per usual. + +When running a workflow by name like this, you get two benefits: + +- **Credentials** are loaded automatically from the credential map in + `openfn.yaml`, so you don't need to pass `--credential-map` +- **Collections** use the server configured in `openfn.yaml`, so you don't need + to pass `--collections-endpoint` or anything. + +## Deploying a Project + +To push your local changes back to the app, run: + +```bash +openfn project deploy +``` + +This will take your currently checked out project and push it to the app. It +will also report what has changed in the local project. + +Before uploading, the CLI fetches the latest version of the project from the app +and checks for **divergence** — meaning it checks whether any of your +locally-changed workflows have also been edited in the app since you last +pulled. If they have, the deploy will fail with an error to prevent you from +accidentally overwriting someone else's work. + +If you want to push anyway, pass `--force`: + +```bash +openfn project deploy --force +``` + +To preview what would change without actually uploading anything, use +`--dry-run`. This will log the final update payload that would be sent to the +app (as a JSON structure). + +You can deploy the checked out project as a new project on the target app by +adding the `--new` flag. This is only available if you have superuser privileges +on the target instance. + +You can also deploy the checked out project to another app project by passing +its alias, id or uuid: + +``` +openfn project deploy main +``` + +If you currently have a development sandbox checked out, this would merge it +straight into the main app project. + +Note that you have to have fetched the target project locally before you can +deploy it. + +## Sandboxes + +The CLI is fully compatible with sandboxes. Treat them like any other project: +fetch them for the first time with their UUID. + +Use the `checkout` command to switch between sandboxes and projects locally. +Remember you can only have a single project checked out at once. The CLI will +warn you if a checkout will cause you to lose any local changes. + +When fetching a sandbox, by default the project alias will be set to the sandbox +name. + +You can merge two projects locally with `openfn project merge`, and deploy the +resulting project to the app (you'll likely have to force push the change). This +is useful for conflict resolution. + +## GitHub + +You can configure a Project to sync automatically to GitHub. This means that +commits to GitHub will automatically deploy changes to an OpenFn Project; and +pressing Save & Sync in the app will commit back to GitHub. + +Under the hood, GitHub Sync uses the CLI's `pull` and `deploy` commands, +triggered by GitHub Actions, to sync your projects. + +Note that by default, GitHub Sync will use the legacy format, with `state.json`, +`project.yaml` and `config.json` files. When setting up a new GitHub Sync, you +can choose to select the v2 format. V2 sync is only suitable for pulling a +single Project per branch to github, as multiple projects will overwrite the +same `workflows/` folder. + +See [Version Control](/documentation/link-to-GitHub) for more details about +GitHub Sync. + +## Cheatsheet + +| Command | Description | +| ---------------------------------------- | ------------------------------------------------------------------------------ | +| `openfn project pull ` | Pull a project from the app for the first time | +| `openfn project pull` | Re-pull the current project | +| `openfn project pull --alias dev` | Pull and set a local alias | +| `openfn project fetch ` | Fetch a project without checking it out | +| `openfn project` | List all local projects in the current working folder | +| `openfn project checkout ` | Switch to a different local project | +| `openfn project deploy` | Deploy checked-out project to the app | +| `openfn project deploy --dry-run` | Try a deploy but skip the upload step | +| `openfn project deploy --force` | Force the checked out project to be uploaded, ignoring any divergence warnings | +| `openfn ` | Run a workflow in the checked-out project | +| `openfn project clean` | Delete the `workflows` folder and all contents, then check out the project | diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index 10f9d2ac8973..cc6d1474526c 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -47,7 +47,7 @@ myProject/ :::info Directory Structure -There are commonly used 3 directory structure for OpenFn projects namely: +There are 3 commonly used directory structures for OpenFn projects, namely: standard, production & test, and monorepo. To learn more, please see the OpenFn [GitHub configuration documentation](/documentation/link-to-GitHub#structuring-your-github-repository). @@ -268,6 +268,16 @@ for resources on a particular Lightning deployment. ## Using the CLI interact with projects +:::warning + +This document describes the legacy OpenFn Sync format used by older versions of +the CLI and GitHub Sync. + +See the [CLI Sync](/documentation/sync) docs for the best experience syncing and +deploying projects. + +::: + The project spec and project state can be used for a variety of reasons, e.g. one could generate the state and spec as backups of the project or one could generate these files and use them for auditing and record keeping, etc. The @@ -395,10 +405,10 @@ OpenFn pull or deploy in your projects. ### Extraneous Workflow ID -#### Description +#### Description This error occurs when you run `openfn deploy` and there is a mismatch between -between IDs of workflows in your projectSpec and your OpenFn instance. When this +IDs of workflows in your projectSpec and your OpenFn instance. When this occurs, the error will be written out in an error object as shown below: ``` @@ -419,6 +429,7 @@ occurs, the error will be written out in an error object as shown below: } } ``` + #### Solution Run `openfn pull` to update your local instance and keep IDs in sync, diff --git a/docs/manage-projects/link-to-gh.md b/docs/manage-projects/link-to-gh.md index 92a8204f03ec..0fd4b84c92e4 100644 --- a/docs/manage-projects/link-to-gh.md +++ b/docs/manage-projects/link-to-gh.md @@ -1,33 +1,29 @@ --- -title: Version Control -sidebar_label: Version Control (GitHub Sync) +title: GitHub Sync +sidebar_label: GitHub Sync slug: /link-to-GitHub --- -The version control (GitHub Sync) feature lets users track and manage changes to -their OpenFn projects in GitHub. GitHub Sync enables a 2-way sync between your -OpenFn project and your GitHub repository. By 2-way sync, we mean that you can -**sync changes made to your project on OpenFn to GitHub**, and you can **deploy -changes you've made to your project on GitHub to OpenFn**. +GitHub Sync enables two-way sync between an OpenFn project and your GitHub +repository. + +This means changes made to a Project in OpenFn can commit back to your GitHub +repo; and commits pushed to GitHub can update the Project in your OpenFn App. :::info For Cloud Hosted OpenFn Users GitHub Sync is only available in projects that are subscribed to Core, Growth, -Scale or Custom plans on Cloud Hosted OpenFn. Users can however connect their -OpenFn account to GitHub by authenticating OpenFn to access their GitHub account -by navigating to `User Profile` page and clicking ` Connect your GitHub -Account". +Scale or Custom plans. ::: -### Configuring your project to use GitHub Sync +### Configuring your Project to use GitHub Sync -Users are able to configure their projects to have access to one or more -repositories on GitHub. To enable sync, the OpenFn project requires a repository -where a GitHub OpenFn application is installed and users are required to have -administrative access to the repositiory. +Users can configure their projects to access one or more repositories on GitHub. +Users must have admin access to the GitHub Repo to ensure the OpenFn application +is installed. -To configure your project to use Github sync, follow the following steps: +To configure your project to use Github sync, follow these steps: 1. Navigate to `Project Settings > Sync to GitHub` . @@ -54,7 +50,7 @@ To configure your project to use Github sync, follow the following steps: ::: 4. Choose your preferred repository and branch you'd like to connect your - project + project to ![Configure](/img/github-options.webp) @@ -83,7 +79,7 @@ To configure your project to use Github sync, follow the following steps: it to GitHub to start the version control process. If you choose to instead take an existing `project.yaml` file from GitHub and overwrite your current OpenFn project, you won't be able to recover your existing workflows on - OpenFn. This is feature that covers certain advanced use-cases, and unless + OpenFn. This is a feature that covers certain advanced use-cases, and unless you know what you're doing you should start by syncing from "OpenFn to GitHub". @@ -113,7 +109,7 @@ interface. After clicking that link, you can follow the steps below: ![Permissions](/img/lightning_gh_permissions.webp) 4. When you're done making changes on GitHub, head back to OpenFn and refresh - the connection lists with the the 🔄 button next to the drop down list of + the connection lists with the 🔄 button next to the drop down list of available installations. ## Using Version Control & Managing Changes @@ -126,7 +122,7 @@ project. ### Sync from OpenFn to GitHub This sync pushes changes from your OpenFn project to GitHub. This sync operation -will trigger a `openfn pull` action workflow on your connected Github repository +will trigger an `openfn pull` action workflow on your connected Github repository , which will pull the latest configuration from the OpenFn app and save it as code in the `project.yaml` file on your repository. @@ -158,7 +154,7 @@ When you trigger `Save & Sync` in a workflow, your new changes and _previous_ uncommitted changes (if any) to your project's resources (including other workflows) will be committed to GitHub. I.e., if there are other, uncommitted changes by either you or someone else to other workflows in the project, they’ll -also show up in that sync also. +also show up in that sync. ::: @@ -185,7 +181,7 @@ to OpenFn. From v2.7.19, OpenFn deploy and pull actions now support the use of relative paths in project spec. Consequently, projects with directory structure that uses relative paths for job code in project spec, automatically gets packaged and -deployed without the user having to copy changes into the projct spec. This new +deployed without the user having to copy changes into the project spec. This new approach gives developers more flexibility to better manage their job code in individual files rather than having all the code in the `projectSpec.yaml` file. @@ -194,8 +190,59 @@ Learn more about relative paths and directory structure in ::: +### Using Sync v2 + +By default, GitHub Sync will use the legacy folder structure to represent your +project on GitHub. That folder structure is explained below - it'll create +`config.json`, `state.json` and `project.yaml` files to represent your Project +in your git repo. + +You can instead choose to use the v2 sync format, as described in the +[Sync](/documentation/sync) pages. This will automatically "expand" your +workflows and steps into files to be easily read and written, and provides a +much better developer experience. + +This v2 style will be made the default means of syncing Projects soon. + +You can also create an empty `openfn.yaml` file in an already connected repo and +the next sync will generate the v2 file structures. + +:::warning + +In Sync v1, you can have multiple 2-way syncs existing on the same branch in a +single repo. That's because each project creates its own set of sync artifacts +(config.json, project.yaml and state.json). Usually you'd want to do this to +sync your production and staging projects, or multiple sandboxes, to the same +GitHub repo. + +This doesn't work with the new sync protocol, because the new sync shares a +`workflows` folder. So each time GitHub pulls from your project, it'll overwrite +`workflows` and wipe out state from your other projects. + +To do this in v2 Sync, you can: + +- Maintain one two-way sync per branch. Each Sandbox maintains its own GitHub + Sync to a different branch in your repo. This works great because you can + compare differences between your sandbox and main project by comparing the + branches on git. +- Connect many Projects to a branch so long as they only sync one-way. This + works in a production environment where one Project is replicated over several + deployments, so committing to GitHub will trigger an update to all connected + Projects. This works so long as you can promise that no user will Save & Sync + back from the production projects + +::: + ## What is in your GitHub Repository? +:::info + +These docs describe the legacy format of GitHub Sync. The latest format is +described in the [CLI Sync](/documentation/sync) pages, and will be used by +default soon. + +::: + When you initiate a connection between OpenFn and your GitHub repository, a config.json file containing reference to your project spec and project state files, and the endpoint of your OpenFn deployment is automatically created in @@ -223,83 +270,6 @@ spec and project state. } ``` -## Structuring OpenFn projects in git repositories - -There are three common patterns used to structure OpenFn projects inside git -repositories. See them below: - -### Standard - -Use this approach if you've got one OpenFn project connected to one git -repository. - -``` -your-git-repo -├── config.json -├── projectState.json -└── projectSpec.yaml -``` - -### Production & Test - -Use this approach if you've got two OpenFn projects that use the _same -worklows_. Here, you're connecting two projects (prod and test) to a single git -repo and a single `project.yaml` file. - -This will allow you to keep two projects in sync when changes are merged from -one branch to another. You might choose to sync: - -- Your production project with the `main` branch -- Your test project with the `staging` branch - -After a merge, your repo would look like this: - -``` -your-git-repo -├── projectSpec.yaml ## works both -│ -├── prod-config.json -├── prod-projectState.json -│ -├── test-config.json -└── test-projectState.json -``` - -### Monorepo - -Sometimes, it's helpful to have multiple OpenFn projects all stored in the same -repo, even if they don't use the same workflows (i.e., even if they don't share -a `project.yaml` file.) - -``` -your-git-monorepo -├── project-a -│ ├── config.json -│ ├── projectState.json -│ └── projectSpec.yaml -└── project-b - ├── config.json - ├── projectState.json - └── projectSpec.yaml -``` - -:::tip A sync in time, saves nine - -#### Syncing Changes from OpenFn to GitHub - -When you sync changes from OpenFn to GitHub, the `projectSpec.yaml` file in your -repository will be updated with the changes made to the project in OpenFn. For a -project with a directory structure that uses relative paths for job code, OpenFn -will respect the structure when syncing changes for backup. All job code will be -written into their respective files. Newer ones will be kept inline (in the -body) of the `projectSpec.yaml` file. - -When you keep job code in relative file paths, ensure to update the -`projectSpec.yaml` file based on changes to the files or paths in your project -repository. A GitHub action is automatically triggered to push changes to OpenFn -ensuring that future syncs are not affected. Changes can include adding, -renaming, deleting a file or updating a file path. ::: - ## Troubleshooting ### Github Sync Error: Unexpected inputs provided: ["snapshots"] diff --git a/sidebars-main.js b/sidebars-main.js index cc0bc31491ca..6cb20f464cbd 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -124,9 +124,10 @@ module.exports = { items: [ 'build-for-developers/cli-intro', 'build-for-developers/cli-usage', + 'build-for-developers/cli-sync', + 'build-for-developers/cli-collections', 'build-for-developers/cli-walkthrough', 'build-for-developers/cli-challenges', - 'build-for-developers/cli-collections', ], }, {