diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..3397dfc --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,94 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## What this is + +DataWeave CLI packages the MuleSoft DataWeave runtime as a **GraalVM native image** so DataWeave scripts run without a JVM. It ships two consumer surfaces from the same runtime: + +- **`dw`** — a native command-line executable (`native-cli`) +- **`dwlib`** — a native shared library with a C-compatible FFI, consumed by Python and Node.js bindings (`native-lib`) + +The DataWeave language/runtime itself lives in a separate repo (`org.mule.weave:*` artifacts, version pinned in `gradle.properties`). This repo is the packaging + CLI layer on top of it. + +## Modules + +- **`native-cli`** — the `dw` executable. Java entrypoint (`org.mule.weave.cli.DWCLI`) using picocli for arg parsing; Scala for the actual command logic (`org.mule.weave.dwnative.*`). `NativeRuntime` wraps `DataWeaveScriptingEngine` and is the heart of script execution. +- **`native-lib`** — the `dwlib` shared library. Java `@CEntryPoint` methods in `org.mule.weave.lib.NativeLib` expose `run_script`, streaming, and callback APIs to non-JVM callers. Contains `python/` and `node/` binding packages. +- **`native-cli-integration-tests`** — runs the DataWeave TCK / weave test suites against the *compiled native binary* (not the JVM). Depends on `native-cli:nativeCompile`. + +## Build & prerequisites + +Requires **GraalVM (Java 24, `graalvm-community`)** with `native-image`. Set `GRAALVM_HOME` and `JAVA_HOME` to it. `./install-graalvm.sh` downloads a distribution into `.graalvm/` and exports the vars (note: the script's pinned layout may lag `graalvmVersion` in `gradle.properties` — verify the path). + +```bash +# Build the dw native executable (~several minutes) → native-cli/build/native/nativeCompile/dw +./gradlew native-cli:nativeCompile + +# Build the shared library → native-lib/build/native/nativeCompile/dwlib.{dylib,so,dll} +./gradlew native-lib:nativeCompile + +# Full build (compiles both native images + runs unit tests) +./gradlew build -PskipNodeTests=true + +# Packaged OS-specific zip distribution of dw +./gradlew native-cli:distro +``` + +`native-lib` native builds use `-J-Xmx6G`; ensure enough memory. + +## Tests + +```bash +# Scala unit tests (JVM, fast) — CLI behavior +./gradlew native-cli:test + +# native-lib Java tests + Python + Node bindings (Node/Python spawn subprocesses) +./gradlew native-lib:test +./gradlew native-lib:pythonTest +./gradlew native-lib:nodeTest + +# Integration tests — run the DataWeave TCK against the compiled dw binary. +# Downloads test-suite zips for the given weave version, then requires native-cli:nativeCompile. +./gradlew -PweaveTestSuiteVersion=2.10.0 -DweaveSuiteVersion=2.10.0 native-cli-integration-tests:test +``` + +Skip flags for CI/local: `-PskipNodeTests=true`, `-PskipPythonTests=true`, `-PskipStripDebug=true` (keep debug symbols in `dwlib`). + +Scala tests use **scalatest** (`AnyFreeSpec` / `Matchers`). Run a single test with the standard scalatest test filter: + +```bash +./gradlew native-cli:test --tests "org.mule.weave.dwnative.cli.DataWeaveCLITest" +``` + +## Version wiring + +Weave runtime and test-suite versions are pinned in `gradle.properties` (`weaveVersion`, `weaveTestSuiteVersion`, `ioVersion`, `weaveSuiteVersion`). At build time `native-cli`'s `genVersions` task generates `org.mule.weave.v2.version.ComponentVersion` (Scala) from these into `build/genresource/` — don't edit it by hand. `nativeVersion` (`100.100.100`) is the CLI/lib artifact version. + +## Native-image gotchas + +The GraalVM `buildArgs` in each `build.gradle` are load-bearing. When adding dependencies that use reflection, threads at startup, or resource loading, you'll likely need to touch these: + +- `--initialize-at-run-time=...` (netty, coursier, `scala.util.Random`, the SPI module loader) — classes that must NOT be initialized during image build. +- **Data formats and module loaders are registered via SPI** (`META-INF/services/org.mule.weave.v2.module.DataFormat` and `...parser.phase.ModuleLoader`). `native-lib` re-materializes these service files during `nativeCompileClasspathJar` (see the `configureEach` block) because the fat-jar merge drops them. A missing/unregistered format shows up as a runtime "unknown mime type" rather than a build error. + +## Bindings (native-lib) + +`dwlib` is staged into the binding packages by Gradle, not committed built: + +- `stagePythonNativeLib` / `stageNodeNativeLib` copy `dwlib.*` into `python/src/dataweave/native/` and `node/native/`. +- `buildPythonWheel` (setup.py `bdist_wheel`) and `buildNodePackage` (npm install → node-gyp rebuild → tsc → npm pack) produce distributables. +- Bindings locate the library via `DATAWEAVE_NATIVE_LIB=/abs/path/to/dwlib.*`, then fall back to the packaged/dev-build locations. + +See `native-lib/README.md` for the full binding API (sync `run`, `run_streaming`, `run_transform`, callback streaming, error types). + +## Domain concepts (`dw` CLI) + +- **Spells** — shareable, runnable DataWeave scripts fetched from git-based "grimoire" repos (`dw spell`, `spell create/list/update`). A grimoire is named `-data-weave-grimoire`; the built-in DW grimoire has no prefix. +- **Wizards** — trusted sources of grimoires (`dw wizard add`). +- **Dependency manager** — a spell's `dependencies.dwl` declares maven deps resolved via coursier (`DependencyManager` / `MavenDependencyManager`). +- Env vars: `DW_HOME` (default `~/.dw`), `DW_DEFAULT_INPUT_MIMETYPE`, `DW_DEFAULT_OUTPUT_MIMETYPE` (both default `application/json`). + +## CI + +`.github/workflows/main.yml` (push/PR to master) and `ci.yml` (weekly) build both native images and all bindings on Ubuntu + Windows. Integration/regression tests (`native-cli-integration-tests` against multiple weave suite versions) run **only on master**, not on PRs, to save CI time. \ No newline at end of file diff --git a/native-lib/build.gradle b/native-lib/build.gradle index d7bbe10..e18568e 100644 --- a/native-lib/build.gradle +++ b/native-lib/build.gradle @@ -67,7 +67,7 @@ graalvmNative { tasks.named('nativeCompile').configure { doLast { if (!System.getProperty('os.name').toLowerCase().contains('windows')) { - def nativeDir = file("${buildDir}/native/nativeCompile") + def nativeDir = file("${layout.buildDirectory.get().asFile}/native/nativeCompile") def dwlibFile = fileTree(nativeDir).matching { include 'dwlib.so', 'dwlib.dylib' }.singleFile if (dwlibFile?.exists()) { def ext = dwlibFile.name.endsWith('.dylib') ? '.dylib' : '.so' @@ -89,7 +89,7 @@ tasks.register('stripNativeLibrary', Exec) { project.findProperty('skipStripDebug')?.toString()?.toBoolean() != true } - def nativeDir = file("${buildDir}/native/nativeCompile") + def nativeDir = file("${layout.buildDirectory.get().asFile}/native/nativeCompile") if (System.getProperty('os.name').toLowerCase().contains('windows')) { // Windows: Use strip from mingw-w64 or MSYS2 if available @@ -117,7 +117,7 @@ def pythonExe = (project.findProperty('pythonExe') ?: 'python3') as String tasks.register('stagePythonNativeLib', Copy) { dependsOn tasks.named('stripNativeLibrary') - from("${buildDir}/native/nativeCompile") { + from("${layout.buildDirectory.get().asFile}/native/nativeCompile") { include('dwlib.*') } into("${projectDir}/python/src/dataweave/native") @@ -158,7 +158,7 @@ tasks.register('pythonTest', Exec) { tasks.register('stageNodeNativeLib', Copy) { dependsOn tasks.named('stripNativeLibrary') - from("${buildDir}/native/nativeCompile") { + from("${layout.buildDirectory.get().asFile}/native/nativeCompile") { include('dwlib.*') } into("${projectDir}/node/native") diff --git a/native-lib/node/.gitignore b/native-lib/node/.gitignore index e76ab8e..b48128c 100644 --- a/native-lib/node/.gitignore +++ b/native-lib/node/.gitignore @@ -2,4 +2,5 @@ node_modules/ dist/ build/ native/ +coverage/ *.tgz diff --git a/native-lib/node/package-lock.json b/native-lib/node/package-lock.json index def3096..727daae 100644 --- a/native-lib/node/package-lock.json +++ b/native-lib/node/package-lock.json @@ -14,6 +14,7 @@ ], "devDependencies": { "@types/node": "^20", + "@vitest/coverage-v8": "^3.0", "node-gyp": "^10", "typescript": "^5.5", "vitest": "^3.0" @@ -22,6 +23,80 @@ "node": ">=18" } }, + "node_modules/@ampproject/remapping": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/@ampproject/remapping/-/remapping-2.3.0.tgz", + "integrity": "sha512-30iZtAPgz+LTIYoeivqYo853f02jBYSd5uGnGpkFV0M3xOt9aN73erkgYAmZU43x4VfqcnLxW9Kpg3R5LC4YYw==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "@jridgewell/gen-mapping": "^0.3.5", + "@jridgewell/trace-mapping": "^0.3.24" + }, + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/@babel/helper-string-parser": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.29.7.tgz", + "integrity": "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/helper-validator-identifier": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.29.7.tgz", + "integrity": "sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@babel/parser": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.7.tgz", + "integrity": "sha512-hnORnjP/1P/zFEndoeX+n+t1RwWRJiJpM/jO7FW32Kn9r5+sJB2JWOdYo4L6k78j15eCwY3Gm/7364B1EMwtNg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/types": "^7.29.7" + }, + "bin": { + "parser": "bin/babel-parser.js" + }, + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/@babel/types": { + "version": "7.29.7", + "resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.7.tgz", + "integrity": "sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/helper-string-parser": "^7.29.7", + "@babel/helper-validator-identifier": "^7.29.7" + }, + "engines": { + "node": ">=6.9.0" + } + }, + "node_modules/@bcoe/v8-coverage": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/@bcoe/v8-coverage/-/v8-coverage-1.0.2.tgz", + "integrity": "sha512-6zABk/ECA/QYSCQ1NGiVwwbQerUCZ+TQbp64Q3AgmfNvurHH0j8TtXa1qbShXA6qqkpAj4V5W8pP6mLe1mcMqA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18" + } + }, "node_modules/@esbuild/aix-ppc64": { "version": "0.25.12", "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.25.12.tgz", @@ -482,6 +557,37 @@ "node": ">=12" } }, + "node_modules/@istanbuljs/schema": { + "version": "0.1.6", + "resolved": "https://registry.npmjs.org/@istanbuljs/schema/-/schema-0.1.6.tgz", + "integrity": "sha512-+Sg6GCR/wy1oSmQDFq4LQDAhm3ETKnorxN+y5nbLULOR3P0c14f2Wurzj3/xqPXtasLFfHd5iRFQ7AJt4KH2cw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/@jridgewell/gen-mapping": { + "version": "0.3.13", + "resolved": "https://registry.npmjs.org/@jridgewell/gen-mapping/-/gen-mapping-0.3.13.tgz", + "integrity": "sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.5.0", + "@jridgewell/trace-mapping": "^0.3.24" + } + }, + "node_modules/@jridgewell/resolve-uri": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz", + "integrity": "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6.0.0" + } + }, "node_modules/@jridgewell/sourcemap-codec": { "version": "1.5.5", "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz", @@ -489,6 +595,17 @@ "dev": true, "license": "MIT" }, + "node_modules/@jridgewell/trace-mapping": { + "version": "0.3.31", + "resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.31.tgz", + "integrity": "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/resolve-uri": "^3.1.0", + "@jridgewell/sourcemap-codec": "^1.4.14" + } + }, "node_modules/@npmcli/agent": { "version": "2.2.2", "resolved": "https://registry.npmjs.org/@npmcli/agent/-/agent-2.2.2.tgz", @@ -915,16 +1032,50 @@ "undici-types": "~6.21.0" } }, + "node_modules/@vitest/coverage-v8": { + "version": "3.2.7", + "resolved": "https://registry.npmjs.org/@vitest/coverage-v8/-/coverage-v8-3.2.7.tgz", + "integrity": "sha512-NEGWJS2XNu2PfRLQwOO3CTKj1tTETxNBdk454vDxVBhxJYhPaA/eS0nAI0c+1El1P7a60z8+i+ZrQoGESweGKg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@ampproject/remapping": "^2.3.0", + "@bcoe/v8-coverage": "^1.0.2", + "ast-v8-to-istanbul": "^0.3.3", + "debug": "^4.4.1", + "istanbul-lib-coverage": "^3.2.2", + "istanbul-lib-report": "^3.0.1", + "istanbul-lib-source-maps": "^5.0.6", + "istanbul-reports": "^3.1.7", + "magic-string": "^0.30.17", + "magicast": "^0.3.5", + "std-env": "^3.9.0", + "test-exclude": "^7.0.1", + "tinyrainbow": "^2.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "@vitest/browser": "3.2.7", + "vitest": "3.2.7" + }, + "peerDependenciesMeta": { + "@vitest/browser": { + "optional": true + } + } + }, "node_modules/@vitest/expect": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.4.tgz", - "integrity": "sha512-Io0yyORnB6sikFlt8QW5K7slY4OjqNX9jmJQ02QDda8lyM6B5oNgVWoSoKPac8/kgnCUzuHQKrSLtu/uOqqrig==", + "version": "3.2.7", + "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.7.tgz", + "integrity": "sha512-E8eBXaKibuvH2pSZErOjdVb5vF4PbKYcrnluBTYxEk1l/VhhwZg1kZQsdtjq+CsF5CFydf2Rdkz7jDHKSisi3w==", "dev": true, "license": "MIT", "dependencies": { "@types/chai": "^5.2.2", - "@vitest/spy": "3.2.4", - "@vitest/utils": "3.2.4", + "@vitest/spy": "3.2.7", + "@vitest/utils": "3.2.7", "chai": "^5.2.0", "tinyrainbow": "^2.0.0" }, @@ -933,13 +1084,13 @@ } }, "node_modules/@vitest/mocker": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-3.2.4.tgz", - "integrity": "sha512-46ryTE9RZO/rfDd7pEqFl7etuyzekzEhUbTW3BvmeO/BcCMEgq59BKhek3dXDWgAj4oMK6OZi+vRr1wPW6qjEQ==", + "version": "3.2.7", + "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-3.2.7.tgz", + "integrity": "sha512-Trr0hYO9CM3Wj6ksWHRhK9IZpIY6wTMO5u/MqXurMxT57sWBaOPEtP3Oq60ihZuh5JsiagKfz95OcxdEP6dBrA==", "dev": true, "license": "MIT", "dependencies": { - "@vitest/spy": "3.2.4", + "@vitest/spy": "3.2.7", "estree-walker": "^3.0.3", "magic-string": "^0.30.17" }, @@ -960,9 +1111,9 @@ } }, "node_modules/@vitest/pretty-format": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.4.tgz", - "integrity": "sha512-IVNZik8IVRJRTr9fxlitMKeJeXFFFN0JaB9PHPGQ8NKQbGpfjlTx9zO4RefN8gp7eqjNy8nyK3NZmBzOPeIxtA==", + "version": "3.2.7", + "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.7.tgz", + "integrity": "sha512-KUHlwqVu0sRlhCdyPdQ/wBoTfRahjUky1MubOmYw9fWfIZy1gNoHpuaaQBPAaMaVYdQYHJLurzj8ECCj5OwTqA==", "dev": true, "license": "MIT", "dependencies": { @@ -973,13 +1124,13 @@ } }, "node_modules/@vitest/runner": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-3.2.4.tgz", - "integrity": "sha512-oukfKT9Mk41LreEW09vt45f8wx7DordoWUZMYdY/cyAk7w5TWkTRCNZYF7sX7n2wB7jyGAl74OxgwhPgKaqDMQ==", + "version": "3.2.7", + "resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-3.2.7.tgz", + "integrity": "sha512-sB9y4ovltoQP+WaUPwmSxO9WIg9Ig694Di5PalVPsYHklAdE027mehpWF2SQSVq+k6sFgaivbTjTJwZLSHbedA==", "dev": true, "license": "MIT", "dependencies": { - "@vitest/utils": "3.2.4", + "@vitest/utils": "3.2.7", "pathe": "^2.0.3", "strip-literal": "^3.0.0" }, @@ -988,13 +1139,13 @@ } }, "node_modules/@vitest/snapshot": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-3.2.4.tgz", - "integrity": "sha512-dEYtS7qQP2CjU27QBC5oUOxLE/v5eLkGqPE0ZKEIDGMs4vKWe7IjgLOeauHsR0D5YuuycGRO5oSRXnwnmA78fQ==", + "version": "3.2.7", + "resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-3.2.7.tgz", + "integrity": "sha512-7C+MwShwtBSI5Buwoyg3s/iY1eHL9PKAf+O1wVh/TdnjXUtkoL/9YQtre90i4MtNXM6edP1wJ2zOBpfCyhIS7g==", "dev": true, "license": "MIT", "dependencies": { - "@vitest/pretty-format": "3.2.4", + "@vitest/pretty-format": "3.2.7", "magic-string": "^0.30.17", "pathe": "^2.0.3" }, @@ -1003,9 +1154,9 @@ } }, "node_modules/@vitest/spy": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-3.2.4.tgz", - "integrity": "sha512-vAfasCOe6AIK70iP5UD11Ac4siNUNJ9i/9PZ3NKx07sG6sUxeag1LWdNrMWeKKYBLlzuK+Gn65Yd5nyL6ds+nw==", + "version": "3.2.7", + "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-3.2.7.tgz", + "integrity": "sha512-Q2eQGI6d2L/hBtZ0qNuKcAGid68XK6cv1xsoaIma6PaJhHPoqcEJhYpXZ/5myCMqkNgtP6UKuBhbc0nHKnrkuQ==", "dev": true, "license": "MIT", "dependencies": { @@ -1016,13 +1167,13 @@ } }, "node_modules/@vitest/utils": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-3.2.4.tgz", - "integrity": "sha512-fB2V0JFrQSMsCo9HiSq3Ezpdv4iYaXRG1Sx8edX3MwxfyNn83mKiGzOcH+Fkxt4MHxr3y42fQi1oeAInqgX2QA==", + "version": "3.2.7", + "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-3.2.7.tgz", + "integrity": "sha512-x6BDOd7dyo3PFLY3I9/HJ25X/6OurhGXk2/B9gOZNPF7XDVjeBK4k01lQE5uvDpbuheErh91qYuE1E2OEjK3Rw==", "dev": true, "license": "MIT", "dependencies": { - "@vitest/pretty-format": "3.2.4", + "@vitest/pretty-format": "3.2.7", "loupe": "^3.1.4", "tinyrainbow": "^2.0.0" }, @@ -1100,6 +1251,18 @@ "node": ">=12" } }, + "node_modules/ast-v8-to-istanbul": { + "version": "0.3.12", + "resolved": "https://registry.npmjs.org/ast-v8-to-istanbul/-/ast-v8-to-istanbul-0.3.12.tgz", + "integrity": "sha512-BRRC8VRZY2R4Z4lFIL35MwNXmwVqBityvOIwETtsCSwvjl0IdgFsy9NhdaA6j74nUdtJJlIypeRhpDam19Wq3g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/trace-mapping": "^0.3.31", + "estree-walker": "^3.0.3", + "js-tokens": "^10.0.0" + } + }, "node_modules/balanced-match": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", @@ -1494,6 +1657,23 @@ "dev": true, "license": "ISC" }, + "node_modules/has-flag": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/has-flag/-/has-flag-4.0.0.tgz", + "integrity": "sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/html-escaper": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/html-escaper/-/html-escaper-2.0.2.tgz", + "integrity": "sha512-H2iMtd0I4Mt5eYiapRdIDjp+XzelXQ0tFE4JS7YFwFevXXMmOp9myNrUvCg0D6ws8iqkRPBfKHgbwig1SmlLfg==", + "dev": true, + "license": "MIT" + }, "node_modules/http-cache-semantics": { "version": "4.2.0", "resolved": "https://registry.npmjs.org/http-cache-semantics/-/http-cache-semantics-4.2.0.tgz", @@ -1600,6 +1780,60 @@ "node": ">=18" } }, + "node_modules/istanbul-lib-coverage": { + "version": "3.2.2", + "resolved": "https://registry.npmjs.org/istanbul-lib-coverage/-/istanbul-lib-coverage-3.2.2.tgz", + "integrity": "sha512-O8dpsF+r0WV/8MNRKfnmrtCWhuKjxrq2w+jpzBL5UZKTi2LeVWnWOmWRxFlesJONmc+wLAGvKQZEOanko0LFTg==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=8" + } + }, + "node_modules/istanbul-lib-report": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/istanbul-lib-report/-/istanbul-lib-report-3.0.1.tgz", + "integrity": "sha512-GCfE1mtsHGOELCU8e/Z7YWzpmybrx/+dSTfLrvY8qRmaY6zXTKWn6WQIjaAFw069icm6GVMNkgu0NzI4iPZUNw==", + "dev": true, + "license": "BSD-3-Clause", + "dependencies": { + "istanbul-lib-coverage": "^3.0.0", + "make-dir": "^4.0.0", + "supports-color": "^7.1.0" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/istanbul-lib-source-maps": { + "version": "5.0.6", + "resolved": "https://registry.npmjs.org/istanbul-lib-source-maps/-/istanbul-lib-source-maps-5.0.6.tgz", + "integrity": "sha512-yg2d+Em4KizZC5niWhQaIomgf5WlL4vOOjZ5xGCmF8SnPE/mDWWXgvRExdcpCgh9lLRRa1/fSYp2ymmbJ1pI+A==", + "dev": true, + "license": "BSD-3-Clause", + "dependencies": { + "@jridgewell/trace-mapping": "^0.3.23", + "debug": "^4.1.1", + "istanbul-lib-coverage": "^3.0.0" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/istanbul-reports": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/istanbul-reports/-/istanbul-reports-3.2.0.tgz", + "integrity": "sha512-HGYWWS/ehqTV3xN10i23tkPkpH46MLCIMFNCaaKNavAXTF1RkqxawEPtnjnGZ6XKSInBKkiOA5BKS+aZiY3AvA==", + "dev": true, + "license": "BSD-3-Clause", + "dependencies": { + "html-escaper": "^2.0.0", + "istanbul-lib-report": "^3.0.0" + }, + "engines": { + "node": ">=8" + } + }, "node_modules/jackspeak": { "version": "3.4.3", "resolved": "https://registry.npmjs.org/jackspeak/-/jackspeak-3.4.3.tgz", @@ -1617,9 +1851,9 @@ } }, "node_modules/js-tokens": { - "version": "9.0.1", - "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-9.0.1.tgz", - "integrity": "sha512-mxa9E9ITFOt0ban3j6L5MpjwegGz6lBQmM1IJkWeBZGcMxto50+eWdjC/52xDbS2vy0k7vIMK0Fe2wfL9OQSpQ==", + "version": "10.0.0", + "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-10.0.0.tgz", + "integrity": "sha512-lM/UBzQmfJRo9ABXbPWemivdCW8V2G8FHaHdypQaIy523snUjog0W71ayWXTjiR+ixeMyVHN2XcpnTd/liPg/Q==", "dev": true, "license": "MIT" }, @@ -1647,6 +1881,34 @@ "@jridgewell/sourcemap-codec": "^1.5.5" } }, + "node_modules/magicast": { + "version": "0.3.5", + "resolved": "https://registry.npmjs.org/magicast/-/magicast-0.3.5.tgz", + "integrity": "sha512-L0WhttDl+2BOsybvEOLK7fW3UA0OQ0IQ2d6Zl2x/a6vVRs3bAY0ECOSHHeL5jD+SbOpOCUEi0y1DgHEn9Qn1AQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@babel/parser": "^7.25.4", + "@babel/types": "^7.25.4", + "source-map-js": "^1.2.0" + } + }, + "node_modules/make-dir": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/make-dir/-/make-dir-4.0.0.tgz", + "integrity": "sha512-hXdUTZYIVOt1Ex//jAQi+wTZZpUpwBj/0QsOzqegb3rGMMeJiSEu5xLHnYfBrRV4RH2+OCSOO95Is/7x1WJ4bw==", + "dev": true, + "license": "MIT", + "dependencies": { + "semver": "^7.5.3" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/make-fetch-happen": { "version": "13.0.1", "resolved": "https://registry.npmjs.org/make-fetch-happen/-/make-fetch-happen-13.0.1.tgz", @@ -2384,6 +2646,26 @@ "url": "https://github.com/sponsors/antfu" } }, + "node_modules/strip-literal/node_modules/js-tokens": { + "version": "9.0.1", + "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-9.0.1.tgz", + "integrity": "sha512-mxa9E9ITFOt0ban3j6L5MpjwegGz6lBQmM1IJkWeBZGcMxto50+eWdjC/52xDbS2vy0k7vIMK0Fe2wfL9OQSpQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/supports-color": { + "version": "7.2.0", + "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz", + "integrity": "sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==", + "dev": true, + "license": "MIT", + "dependencies": { + "has-flag": "^4.0.0" + }, + "engines": { + "node": ">=8" + } + }, "node_modules/tar": { "version": "6.2.1", "resolved": "https://registry.npmjs.org/tar/-/tar-6.2.1.tgz", @@ -2439,6 +2721,60 @@ "node": ">=8" } }, + "node_modules/test-exclude": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/test-exclude/-/test-exclude-7.0.2.tgz", + "integrity": "sha512-u9E6A+ZDYdp7a4WnarkXPZOx8Ilz46+kby6p1yZ8zsGTz9gYa6FIS7lj2oezzNKmtdyyJNNmmXDppga5GB7kSw==", + "dev": true, + "license": "ISC", + "dependencies": { + "@istanbuljs/schema": "^0.1.2", + "glob": "^10.4.1", + "minimatch": "^10.2.2" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/test-exclude/node_modules/balanced-match": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz", + "integrity": "sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==", + "dev": true, + "license": "MIT", + "engines": { + "node": "18 || 20 || >=22" + } + }, + "node_modules/test-exclude/node_modules/brace-expansion": { + "version": "5.0.7", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.7.tgz", + "integrity": "sha512-7oFy703dxfY3/NLxC1fh2SUCQ0H9rmAY+5EpDVfXjUTTs+HEwR2nYaqLv+GWcTsumwxPfiz6CzCNkwXwBUwqCA==", + "dev": true, + "license": "MIT", + "dependencies": { + "balanced-match": "^4.0.2" + }, + "engines": { + "node": "18 || 20 || >=22" + } + }, + "node_modules/test-exclude/node_modules/minimatch": { + "version": "10.2.5", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.5.tgz", + "integrity": "sha512-MULkVLfKGYDFYejP07QOurDLLQpcjk7Fw+7jXS2R2czRQzR56yHRveU5NDJEOviH+hETZKSkIk5c+T23GjFUMg==", + "dev": true, + "license": "BlueOak-1.0.0", + "dependencies": { + "brace-expansion": "^5.0.5" + }, + "engines": { + "node": "18 || 20 || >=22" + }, + "funding": { + "url": "https://github.com/sponsors/isaacs" + } + }, "node_modules/tinybench": { "version": "2.9.0", "resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz", @@ -2646,20 +2982,20 @@ } }, "node_modules/vitest": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/vitest/-/vitest-3.2.4.tgz", - "integrity": "sha512-LUCP5ev3GURDysTWiP47wRRUpLKMOfPh+yKTx3kVIEiu5KOMeqzpnYNsKyOoVrULivR8tLcks4+lga33Whn90A==", + "version": "3.2.7", + "resolved": "https://registry.npmjs.org/vitest/-/vitest-3.2.7.tgz", + "integrity": "sha512-KrxIJ62Fd89gfysR4WotlgZABiz2dqFPgqGzX7s+CwsqLFomRH7777ZcrOD6+WVAh7khPQP41A+BKbpcJFrdEg==", "dev": true, "license": "MIT", "dependencies": { "@types/chai": "^5.2.2", - "@vitest/expect": "3.2.4", - "@vitest/mocker": "3.2.4", - "@vitest/pretty-format": "^3.2.4", - "@vitest/runner": "3.2.4", - "@vitest/snapshot": "3.2.4", - "@vitest/spy": "3.2.4", - "@vitest/utils": "3.2.4", + "@vitest/expect": "3.2.7", + "@vitest/mocker": "3.2.7", + "@vitest/pretty-format": "^3.2.7", + "@vitest/runner": "3.2.7", + "@vitest/snapshot": "3.2.7", + "@vitest/spy": "3.2.7", + "@vitest/utils": "3.2.7", "chai": "^5.2.0", "debug": "^4.4.1", "expect-type": "^1.2.1", @@ -2689,8 +3025,8 @@ "@edge-runtime/vm": "*", "@types/debug": "^4.1.12", "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", - "@vitest/browser": "3.2.4", - "@vitest/ui": "3.2.4", + "@vitest/browser": "3.2.7", + "@vitest/ui": "3.2.7", "happy-dom": "*", "jsdom": "*" }, diff --git a/native-lib/node/package.json b/native-lib/node/package.json index cf49228..9d4da40 100644 --- a/native-lib/node/package.json +++ b/native-lib/node/package.json @@ -10,6 +10,11 @@ "build:ts": "tsc", "test": "vitest run", "test:watch": "vitest", + "test:unit": "vitest run --project unit", + "test:integration": "vitest run --project integration", + "test:tck": "vitest run --project tck", + "test:coverage": "vitest run --coverage", + "test:coverage:unit": "vitest run --project unit --coverage", "pack": "npm pack" }, "files": [ @@ -31,6 +36,7 @@ "dependencies": {}, "devDependencies": { "@types/node": "^20", + "@vitest/coverage-v8": "^3.0", "node-gyp": "^10", "typescript": "^5.5", "vitest": "^3.0" diff --git a/native-lib/node/src/index.ts b/native-lib/node/src/index.ts index 2902003..762ab35 100644 --- a/native-lib/node/src/index.ts +++ b/native-lib/node/src/index.ts @@ -1,5 +1,6 @@ import * as ffi from "./ffi"; import { findLibrary, buildInputsJson } from "./utils"; +import { parseNativeResponse, parseStreamingResult } from "./result"; import type { ExecutionResult, StreamingResult, @@ -32,82 +33,6 @@ export class DataWeaveScriptError extends DataWeaveError { } } -function parseNativeResponse(raw: string): ExecutionResult { - if (!raw) { - return makeResult(false, null, "Native returned empty response", false, null, null); - } - - let parsed: Record; - try { - parsed = JSON.parse(raw); - } catch (e) { - return makeResult(false, null, `Failed to parse native JSON response: ${e}`, false, null, null); - } - - const success = Boolean(parsed.success); - if (!success) { - return makeResult(false, null, (parsed.error as string) ?? null, false, null, null); - } - - return makeResult( - true, - (parsed.result as string) ?? null, - null, - Boolean(parsed.binary), - (parsed.mimeType as string) ?? null, - (parsed.charset as string) ?? null - ); -} - -function makeResult( - success: boolean, - result: string | null, - error: string | null, - binary: boolean, - mimeType: string | null, - charset: string | null -): ExecutionResult { - return { - success, - result, - error, - binary, - mimeType, - charset, - getBytes() { - if (!this.success || this.result === null) return null; - return Buffer.from(this.result, "base64"); - }, - getString() { - if (!this.success || this.result === null) return null; - if (this.binary) return this.result; - const bytes = Buffer.from(this.result, "base64"); - return bytes.toString((this.charset as BufferEncoding) ?? "utf-8"); - }, - }; -} - -function parseStreamingResult(raw: string): StreamingResult { - let meta: Record; - try { - meta = raw ? JSON.parse(raw) : { success: false, error: "Empty response" }; - } catch { - return { success: false, error: "Failed to parse metadata", mimeType: null, charset: null, binary: false }; - } - - const success = Boolean(meta.success); - if (!success) { - return { success: false, error: (meta.error as string) ?? null, mimeType: null, charset: null, binary: false }; - } - return { - success: true, - error: null, - mimeType: (meta.mimeType as string) ?? null, - charset: (meta.charset as string) ?? null, - binary: Boolean(meta.binary), - }; -} - export class DataWeave { private libPath: string; private initialized = false; diff --git a/native-lib/node/src/result.ts b/native-lib/node/src/result.ts new file mode 100644 index 0000000..267fe2f --- /dev/null +++ b/native-lib/node/src/result.ts @@ -0,0 +1,122 @@ +import type { ExecutionResult, StreamingResult } from "./types"; + +/** + * Parses the JSON envelope returned by the native `run_script` FFI call into an + * {@link ExecutionResult}. + * + * The native layer returns a JSON object of the shape + * `{ success, result, error, binary, mimeType, charset }`, where `result` is a + * base64-encoded payload. Any failure to obtain that envelope — an empty + * string, malformed JSON, or `success: false` — is surfaced as an + * unsuccessful result rather than a thrown error, so callers can branch on + * `result.success` uniformly. + * + * @param raw - The raw JSON string produced by the native call. + * @returns A successful result carrying the decoded payload, or an + * unsuccessful result whose `error` describes what went wrong. + */ +export function parseNativeResponse(raw: string): ExecutionResult { + if (!raw) { + return makeResult(false, null, "Native returned empty response", false, null, null); + } + + let parsed: Record; + try { + parsed = JSON.parse(raw); + } catch (e) { + return makeResult(false, null, `Failed to parse native JSON response: ${e}`, false, null, null); + } + + const success = Boolean(parsed.success); + if (!success) { + return makeResult(false, null, (parsed.error as string) ?? null, false, null, null); + } + + return makeResult( + true, + (parsed.result as string) ?? null, + null, + Boolean(parsed.binary), + (parsed.mimeType as string) ?? null, + (parsed.charset as string) ?? null + ); +} + +/** + * Builds an {@link ExecutionResult}, attaching the `getBytes` / `getString` + * accessors that decode the base64 `result` payload on demand. + * + * `getBytes` returns the raw decoded bytes as a {@link Buffer}; `getString` + * returns text, decoding binary payloads as-is and non-binary payloads using + * `charset` (defaulting to UTF-8). Both accessors return `null` when the result + * is unsuccessful or carries no payload. + * + * @param success - Whether the script executed successfully. + * @param result - The base64-encoded payload, or `null` when there is none. + * @param error - An error message when unsuccessful, otherwise `null`. + * @param binary - Whether the payload is binary rather than text. + * @param mimeType - The output MIME type, if reported by the runtime. + * @param charset - The output charset used to decode text, if reported. + * @returns The assembled result object with lazy decoding accessors. + */ +export function makeResult( + success: boolean, + result: string | null, + error: string | null, + binary: boolean, + mimeType: string | null, + charset: string | null +): ExecutionResult { + return { + success, + result, + error, + binary, + mimeType, + charset, + getBytes() { + if (!this.success || this.result === null) return null; + return Buffer.from(this.result, "base64"); + }, + getString() { + if (!this.success || this.result === null) return null; + if (this.binary) return this.result; + const bytes = Buffer.from(this.result, "base64"); + return bytes.toString((this.charset as BufferEncoding) ?? "utf-8"); + }, + }; +} + +/** + * Parses the trailing metadata envelope emitted by the streaming FFI calls + * (`run_script_streaming` / `run_script_transform`) into a + * {@link StreamingResult}. + * + * Unlike {@link parseNativeResponse}, the payload itself is delivered + * out-of-band through the chunk callbacks; this envelope carries only the + * final status and content metadata (`mimeType`, `charset`, `binary`). An empty + * string, malformed JSON, or `success: false` all yield an unsuccessful result. + * + * @param raw - The raw JSON metadata string produced once streaming completes. + * @returns The streaming outcome and content metadata. + */ +export function parseStreamingResult(raw: string): StreamingResult { + let meta: Record; + try { + meta = raw ? JSON.parse(raw) : { success: false, error: "Empty response" }; + } catch { + return { success: false, error: "Failed to parse metadata", mimeType: null, charset: null, binary: false }; + } + + const success = Boolean(meta.success); + if (!success) { + return { success: false, error: (meta.error as string) ?? null, mimeType: null, charset: null, binary: false }; + } + return { + success: true, + error: null, + mimeType: (meta.mimeType as string) ?? null, + charset: (meta.charset as string) ?? null, + binary: Boolean(meta.binary), + }; +} \ No newline at end of file diff --git a/native-lib/node/src/types.ts b/native-lib/node/src/types.ts index eec8ee3..1f99a52 100644 --- a/native-lib/node/src/types.ts +++ b/native-lib/node/src/types.ts @@ -1,41 +1,81 @@ +/** + * The outcome of a non-streaming script execution. + * + * The payload is carried in `result` as a base64-encoded string; use the + * {@link ExecutionResult.getBytes} / {@link ExecutionResult.getString} + * accessors to decode it rather than reading `result` directly. + */ export interface ExecutionResult { + /** Whether the script executed successfully. */ success: boolean; + /** The base64-encoded output payload, or `null` when there is none. */ result: string | null; + /** The error message when `success` is `false`, otherwise `null`. */ error: string | null; + /** Whether the payload is binary rather than text. */ binary: boolean; + /** The output MIME type, if reported by the runtime. */ mimeType: string | null; + /** The output charset used to decode text, if reported by the runtime. */ charset: string | null; + /** Decodes and returns the payload as raw bytes, or `null` if unsuccessful/empty. */ getBytes(): Buffer | null; + /** Decodes and returns the payload as a string (using `charset`, default UTF-8), or `null` if unsuccessful/empty. */ getString(): string | null; } +/** + * The terminal metadata of a streaming execution. The output bytes are + * delivered separately via the stream's chunk callbacks; this object carries + * only the final status and content metadata. + */ export interface StreamingResult { + /** Whether the stream completed successfully. */ success: boolean; + /** The error message when `success` is `false`, otherwise `null`. */ error: string | null; + /** The output MIME type, if reported by the runtime. */ mimeType: string | null; + /** The output charset, if reported by the runtime. */ charset: string | null; + /** Whether the streamed payload is binary rather than text. */ binary: boolean; } +/** + * An explicit input value with its content and format metadata. Use this shape + * (rather than a bare primitive) when you need to control the MIME type, + * charset, or reader properties for an input. + */ export interface InputValue { + /** The input content — a string, or a {@link Buffer} for binary data. */ content: string | Buffer; + /** The MIME type of the content (e.g. `application/json`, `text/csv`). */ mimeType: string; + /** The charset used to encode string content; defaults to `utf-8`. */ charset?: string; + /** Optional reader/format properties forwarded to the DataWeave data format. */ properties?: Record; } +/** + * A single named input. Either an explicit {@link InputValue}, or a plain value + * (string, number, boolean, `null`, or object/array) whose MIME type is + * inferred during normalization. + */ export type InputEntry = InputValue | string | number | boolean | null | object; +/** A map of input names (e.g. `payload`) to their values, made available to the script. */ export type Inputs = Record; +/** Options controlling how the primary streaming input of `runTransform` is interpreted. */ export interface TransformOptions { + /** The name the primary input is bound to in the script; defaults to `payload`. */ inputName?: string; + /** The MIME type of the primary input; defaults to `application/json`. */ mimeType?: string; + /** The charset of the primary input, if it is text. */ charset?: string; + /** Additional named inputs made available alongside the primary streaming input. */ inputs?: Inputs; } - -export interface StreamOutput { - stream: AsyncGenerator; - metadata: Promise; -} diff --git a/native-lib/node/src/utils.ts b/native-lib/node/src/utils.ts index 0e3ab0d..6b4bf95 100644 --- a/native-lib/node/src/utils.ts +++ b/native-lib/node/src/utils.ts @@ -2,14 +2,30 @@ import { existsSync } from "node:fs"; import { join, dirname } from "node:path"; import type { InputEntry, Inputs } from "./types"; +/** Environment variable holding an explicit absolute path to the `dwlib` shared library. */ const ENV_NATIVE_LIB = "DATAWEAVE_NATIVE_LIB"; +/** Platform-specific shared-library extensions, in the order they are probed (macOS, Linux, Windows). */ const LIB_EXTENSIONS = [".dylib", ".so", ".dll"]; +/** Returns the candidate `dwlib` file names for every supported platform (`dwlib.dylib`, `dwlib.so`, `dwlib.dll`). */ function libNames(): string[] { return LIB_EXTENSIONS.map((ext) => `dwlib${ext}`); } +/** + * Locates the DataWeave native shared library (`dwlib.*`) on disk. + * + * Resolution is attempted in priority order: + * 1. The {@link ENV_NATIVE_LIB} environment variable, if it points at an existing file. + * 2. The packaged location — `/native/dwlib.*` relative to this module. + * 3. A dev-build fallback — walking up to 10 parent directories looking for + * `build/native/nativeCompile/dwlib.*`. + * 4. The current working directory. + * + * @returns The absolute path to the located library. + * @throws Error if no library can be found in any of the above locations. + */ export function findLibrary(): string { const envValue = (process.env[ENV_NATIVE_LIB] ?? "").trim(); if (envValue && existsSync(envValue)) { @@ -51,6 +67,25 @@ export function findLibrary(): string { ); } +/** + * Normalizes a single input value into the base64-encoded envelope the native + * layer expects: `{ content, mimeType, charset, properties? }`. + * + * The shape is inferred from the value: + * - `null` / `undefined` → the JSON literal `null` as `application/json`. + * - An explicit {@link InputValue} object (has both `content` and `mimeType`) → + * its content is base64-encoded (Buffers directly, strings via `charset`), + * preserving any `charset` and `properties`. + * - A string → `text/plain`. + * - A number, boolean, or any other object/array → JSON-serialized as + * `application/json`, falling back to `String(value)` / `text/plain` if it + * cannot be serialized. + * + * @param value - The raw input entry to normalize. + * @param mimeType - Optional MIME type override; when omitted a type is inferred + * from the value. + * @returns The native-ready envelope with base64-encoded `content`. + */ export function normalizeInputValue(value: InputEntry, mimeType?: string): Record { if (value === null || value === undefined) { const content = Buffer.from("null", "utf-8").toString("base64"); @@ -102,6 +137,13 @@ export function normalizeInputValue(value: InputEntry, mimeType?: string): Recor return { content: encodedContent, mimeType: mimeType ?? defaultMime, charset }; } +/** + * Normalizes every entry in an {@link Inputs} map via {@link normalizeInputValue} + * and serializes the result to the JSON string passed to the native FFI calls. + * + * @param inputs - The named inputs to make available to the script (e.g. `payload`). + * @returns A JSON string mapping each input name to its base64-encoded envelope. + */ export function buildInputsJson(inputs: Inputs): string { const normalized: Record = {}; for (const [key, val] of Object.entries(inputs)) { diff --git a/native-lib/node/tests/dataweave.test.ts b/native-lib/node/tests/integration/dataweave.test.ts similarity index 97% rename from native-lib/node/tests/dataweave.test.ts rename to native-lib/node/tests/integration/dataweave.test.ts index 2c04455..bacf160 100644 --- a/native-lib/node/tests/dataweave.test.ts +++ b/native-lib/node/tests/integration/dataweave.test.ts @@ -1,7 +1,7 @@ import { describe, it, expect, afterAll } from "vitest"; import { readFileSync } from "node:fs"; import { join } from "node:path"; -import { DataWeave, run, runStreaming, runTransform, cleanup } from "../src/index"; +import { DataWeave, run, runStreaming, runTransform, cleanup } from "../../src/index"; afterAll(() => { cleanup(); @@ -35,7 +35,7 @@ describe("DataWeave Node.js API", () => { }); it("encoding: UTF-16 XML to CSV", () => { - const xmlPath = join(__dirname, "fixtures", "person.xml"); + const xmlPath = join(__dirname, "..", "fixtures", "person.xml"); const xmlBytes = readFileSync(xmlPath); const script = `output application/csv header=true @@ -194,7 +194,7 @@ describe("DataWeave Node.js API", () => { }); it("file-based streaming input", async () => { - const xmlPath = join(__dirname, "fixtures", "person.xml"); + const xmlPath = join(__dirname, "..", "fixtures", "person.xml"); const xmlData = readFileSync(xmlPath); function* chunked(data: Buffer, size = 4096): Generator { diff --git a/native-lib/node/tests/tck/.gitkeep b/native-lib/node/tests/tck/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/native-lib/node/tests/unit/result.test.ts b/native-lib/node/tests/unit/result.test.ts new file mode 100644 index 0000000..4d9d6d6 --- /dev/null +++ b/native-lib/node/tests/unit/result.test.ts @@ -0,0 +1,139 @@ +import { describe, it, expect } from "vitest"; +import { parseNativeResponse, makeResult, parseStreamingResult } from "../../src/result"; + +const b64 = (s: string, enc: BufferEncoding = "utf-8") => + Buffer.from(s, enc).toString("base64"); + +describe("parseNativeResponse", () => { + it("treats an empty string as a failure", () => { + const r = parseNativeResponse(""); + expect(r.success).toBe(false); + expect(r.error).toBe("Native returned empty response"); + expect(r.getString()).toBeNull(); + expect(r.getBytes()).toBeNull(); + }); + + it("treats malformed JSON as a failure", () => { + const r = parseNativeResponse("{not json"); + expect(r.success).toBe(false); + expect(r.error).toContain("Failed to parse native JSON response"); + }); + + it("maps an error envelope to a failed result", () => { + const r = parseNativeResponse(JSON.stringify({ success: false, error: "boom" })); + expect(r.success).toBe(false); + expect(r.error).toBe("boom"); + }); + + it("defaults error to null when a failure omits it", () => { + const r = parseNativeResponse(JSON.stringify({ success: false })); + expect(r.success).toBe(false); + expect(r.error).toBeNull(); + }); + + it("maps a success envelope with metadata", () => { + const r = parseNativeResponse( + JSON.stringify({ + success: true, + result: b64("42"), + binary: false, + mimeType: "application/json", + charset: "utf-8", + }) + ); + expect(r.success).toBe(true); + expect(r.error).toBeNull(); + expect(r.mimeType).toBe("application/json"); + expect(r.charset).toBe("utf-8"); + expect(r.binary).toBe(false); + expect(r.getString()).toBe("42"); + }); + + it("defaults optional success fields when absent", () => { + const r = parseNativeResponse(JSON.stringify({ success: true, result: b64("x") })); + expect(r.success).toBe(true); + expect(r.binary).toBe(false); + expect(r.mimeType).toBeNull(); + expect(r.charset).toBeNull(); + }); +}); + +describe("makeResult", () => { + it("getString/getBytes return null on failure", () => { + const r = makeResult(false, null, "err", false, null, null); + expect(r.getString()).toBeNull(); + expect(r.getBytes()).toBeNull(); + }); + + it("getString/getBytes return null when result is null even on success", () => { + const r = makeResult(true, null, null, false, null, null); + expect(r.getString()).toBeNull(); + expect(r.getBytes()).toBeNull(); + }); + + it("getBytes base64-decodes into a Buffer", () => { + const r = makeResult(true, b64("hello"), null, false, null, null); + const bytes = r.getBytes(); + expect(Buffer.isBuffer(bytes)).toBe(true); + expect(bytes!.toString("utf-8")).toBe("hello"); + }); + + it("getString decodes using the declared charset", () => { + const r = makeResult(true, b64("café ☕", "utf-16le"), null, false, null, "utf-16le"); + expect(r.getString()).toBe("café ☕"); + }); + + it("getString defaults to utf-8 when charset is null", () => { + const r = makeResult(true, b64("café"), null, false, null, null); + expect(r.getString()).toBe("café"); + }); + + it("getString passes the raw result through when binary", () => { + const raw = b64("anything"); + const r = makeResult(true, raw, null, true, "application/octet-stream", null); + // binary results are not decoded — the base64 payload is returned as-is + expect(r.getString()).toBe(raw); + // getBytes still decodes the base64 payload + expect(r.getBytes()!.toString("utf-8")).toBe("anything"); + }); +}); + +describe("parseStreamingResult", () => { + it("treats an empty string as a failure", () => { + const r = parseStreamingResult(""); + expect(r.success).toBe(false); + expect(r.error).toBe("Empty response"); + }); + + it("treats malformed JSON as a failure", () => { + const r = parseStreamingResult("{oops"); + expect(r.success).toBe(false); + expect(r.error).toBe("Failed to parse metadata"); + expect(r.mimeType).toBeNull(); + expect(r.charset).toBeNull(); + expect(r.binary).toBe(false); + }); + + it("maps an error envelope to a failed result", () => { + const r = parseStreamingResult(JSON.stringify({ success: false, error: "nope" })); + expect(r.success).toBe(false); + expect(r.error).toBe("nope"); + }); + + it("defaults error to null when a failure omits it", () => { + const r = parseStreamingResult(JSON.stringify({ success: false })); + expect(r.success).toBe(false); + expect(r.error).toBeNull(); + }); + + it("maps a success envelope with metadata", () => { + const r = parseStreamingResult( + JSON.stringify({ success: true, mimeType: "application/csv", charset: "utf-8", binary: true }) + ); + expect(r.success).toBe(true); + expect(r.error).toBeNull(); + expect(r.mimeType).toBe("application/csv"); + expect(r.charset).toBe("utf-8"); + expect(r.binary).toBe(true); + }); +}); diff --git a/native-lib/node/tests/unit/utils.test.ts b/native-lib/node/tests/unit/utils.test.ts new file mode 100644 index 0000000..9d40ab0 --- /dev/null +++ b/native-lib/node/tests/unit/utils.test.ts @@ -0,0 +1,147 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; +import { existsSync } from "node:fs"; +import { normalizeInputValue, buildInputsJson, findLibrary } from "../../src/utils"; + +vi.mock("node:fs", () => ({ existsSync: vi.fn() })); + +const mockedExistsSync = vi.mocked(existsSync); + +const decode = (o: Record) => + Buffer.from(o.content as string, "base64").toString("utf-8"); + +describe("normalizeInputValue", () => { + it("encodes null as JSON null", () => { + const n = normalizeInputValue(null); + expect(n.mimeType).toBe("application/json"); + expect(n.charset).toBe("utf-8"); + expect(decode(n)).toBe("null"); + }); + + it("honors an explicit mimeType for null", () => { + const n = normalizeInputValue(null, "text/plain"); + expect(n.mimeType).toBe("text/plain"); + }); + + it("encodes a string as text/plain", () => { + const n = normalizeInputValue("hello"); + expect(n.mimeType).toBe("text/plain"); + expect(decode(n)).toBe("hello"); + }); + + it("encodes a number as JSON", () => { + const n = normalizeInputValue(42); + expect(n.mimeType).toBe("application/json"); + expect(decode(n)).toBe("42"); + }); + + it("encodes a boolean as JSON", () => { + const n = normalizeInputValue(true); + expect(n.mimeType).toBe("application/json"); + expect(decode(n)).toBe("true"); + }); + + it("encodes an array as JSON", () => { + const n = normalizeInputValue([1, 2, 3]); + expect(n.mimeType).toBe("application/json"); + expect(decode(n)).toBe("[1,2,3]"); + }); + + it("encodes a plain object as JSON", () => { + const n = normalizeInputValue({ a: 1 }); + expect(n.mimeType).toBe("application/json"); + expect(decode(n)).toBe('{"a":1}'); + }); + + it("respects an explicit override mimeType for plain values", () => { + const n = normalizeInputValue("a,b,c", "application/csv"); + expect(n.mimeType).toBe("application/csv"); + expect(decode(n)).toBe("a,b,c"); + }); + + describe("explicit InputValue objects ({content, mimeType})", () => { + it("base64-encodes string content using the declared charset", () => { + const n = normalizeInputValue({ content: "café", mimeType: "text/plain", charset: "utf-8" }); + expect(n.mimeType).toBe("text/plain"); + expect(n.charset).toBe("utf-8"); + expect(decode(n)).toBe("café"); + }); + + it("base64-encodes Buffer content directly", () => { + const buf = Buffer.from("binary-bytes"); + const n = normalizeInputValue({ content: buf, mimeType: "application/octet-stream" }); + expect(n.content).toBe(buf.toString("base64")); + }); + + it("passes through properties and charset", () => { + const n = normalizeInputValue({ + content: "1;2;3", + mimeType: "application/csv", + charset: "utf-8", + properties: { separator: ";", header: false }, + }); + expect(n.charset).toBe("utf-8"); + expect(n.properties).toEqual({ separator: ";", header: false }); + }); + + it("omits charset/properties when not provided", () => { + const n = normalizeInputValue({ content: "x", mimeType: "text/plain" }); + expect("charset" in n).toBe(false); + expect("properties" in n).toBe(false); + }); + }); +}); + +describe("buildInputsJson", () => { + it("normalizes every entry into a JSON string", () => { + const json = buildInputsJson({ num1: 25, name: "bob" }); + const parsed = JSON.parse(json); + expect(decode(parsed.num1)).toBe("25"); + expect(parsed.num1.mimeType).toBe("application/json"); + expect(decode(parsed.name)).toBe("bob"); + expect(parsed.name.mimeType).toBe("text/plain"); + }); + + it("produces an empty object for no inputs", () => { + expect(buildInputsJson({})).toBe("{}"); + }); +}); + +describe("findLibrary", () => { + const ORIGINAL_ENV = process.env.DATAWEAVE_NATIVE_LIB; + + beforeEach(() => { + mockedExistsSync.mockReset(); + delete process.env.DATAWEAVE_NATIVE_LIB; + }); + + afterEach(() => { + if (ORIGINAL_ENV === undefined) delete process.env.DATAWEAVE_NATIVE_LIB; + else process.env.DATAWEAVE_NATIVE_LIB = ORIGINAL_ENV; + }); + + it("returns the DATAWEAVE_NATIVE_LIB path when it exists", () => { + process.env.DATAWEAVE_NATIVE_LIB = "/custom/dwlib.dylib"; + mockedExistsSync.mockImplementation((p) => p === "/custom/dwlib.dylib"); + expect(findLibrary()).toBe("/custom/dwlib.dylib"); + }); + + it("ignores the env var when the path does not exist and falls through", () => { + process.env.DATAWEAVE_NATIVE_LIB = "/missing/dwlib.so"; + // env path missing; first existing candidate is the packaged native/ dir + mockedExistsSync.mockImplementation((p) => String(p).includes("native") && !String(p).includes("/missing/")); + const result = findLibrary(); + expect(result).not.toBe("/missing/dwlib.so"); + expect(result).toContain("dwlib."); + }); + + it("resolves the packaged native/ library when present", () => { + mockedExistsSync.mockImplementation((p) => /native[\\/]dwlib\.(dylib|so|dll)$/.test(String(p))); + expect(findLibrary()).toMatch(/native[\\/]dwlib\.(dylib|so|dll)$/); + }); + + it("throws a helpful error when no library can be located", () => { + mockedExistsSync.mockReturnValue(false); + expect(() => findLibrary()).toThrow(/Could not find DataWeave native library/); + expect(() => findLibrary()).toThrow(/DATAWEAVE_NATIVE_LIB/); + }); +}); diff --git a/native-lib/node/vitest.config.ts b/native-lib/node/vitest.config.ts index 4870abb..5cf67d1 100644 --- a/native-lib/node/vitest.config.ts +++ b/native-lib/node/vitest.config.ts @@ -2,6 +2,43 @@ import { defineConfig } from "vitest/config"; export default defineConfig({ test: { - testTimeout: 30000, + // Groups are built out incrementally, so a lane with no test files yet + // (e.g. tck) must not fail the run. + passWithNoTests: true, + // Three test groups. Run all with `vitest run`, or one lane with + // `vitest run --project unit` (etc.). Coverage merges across whichever ran. + // - unit: pure TS logic, no native library (dwlib) required. + // - integration: exercises the real dwlib via the native addon. + // - tck: DataWeave runtime conformance suite against dwlib. + projects: [ + { + test: { + name: "unit", + include: ["tests/unit/**/*.test.ts"], + testTimeout: 10000, + }, + }, + { + test: { + name: "integration", + include: ["tests/integration/**/*.test.ts"], + testTimeout: 30000, + }, + }, + { + test: { + name: "tck", + include: ["tests/tck/**/*.test.ts"], + testTimeout: 30000, + }, + }, + ], + coverage: { + provider: "v8", + include: ["src/**/*.ts"], + // ffi.ts is a thin require() wrapper over the native addon; nothing to cover. + exclude: ["src/ffi.ts"], + reporter: ["text", "lcov", "html"], + }, }, -}); +}); \ No newline at end of file