From f0f78c29b6c1c492ffba228b5543e48c751cdebc Mon Sep 17 00:00:00 2001 From: Haechan Date: Fri, 11 Sep 2026 06:41:14 +0000 Subject: [PATCH 1/5] Add resumable DART dataset lifecycle evidence harness --- scripts/year3-dart100/README.md | 42 ++++ .../year3-dart100/ainize-inference-audit.js | 180 ++++++++++++++++++ .../year3-dart100/ainize-lifecycle-state.js | 87 +++++++++ scripts/year3-dart100/ainize-lifecycle.js | 124 ++++++++++++ scripts/year3-dart100/m5-judge.js | 34 ++++ scripts/year3-dart100/package.json | 11 ++ .../test/ainize-lifecycle.test.js | 136 +++++++++++++ 7 files changed, 614 insertions(+) create mode 100644 scripts/year3-dart100/README.md create mode 100644 scripts/year3-dart100/ainize-inference-audit.js create mode 100644 scripts/year3-dart100/ainize-lifecycle-state.js create mode 100644 scripts/year3-dart100/ainize-lifecycle.js create mode 100644 scripts/year3-dart100/m5-judge.js create mode 100644 scripts/year3-dart100/package.json create mode 100644 scripts/year3-dart100/test/ainize-lifecycle.test.js diff --git a/scripts/year3-dart100/README.md b/scripts/year3-dart100/README.md new file mode 100644 index 0000000..c1521fa --- /dev/null +++ b/scripts/year3-dart100/README.md @@ -0,0 +1,42 @@ +# DART100 lifecycle observation harness + +Experimental, deployment-specific evidence tooling. It does not claim 100 distinct models, public Ainize listings, HF model deployment, or a completed year-three performance evaluation. + +## Scope + +The existing deployment has 100 registered DART task datasets (780 canonical rows) and a public Hugging Face dataset repository, `Minhyun/ainize-dart100-reproduction-20260911`, at revision `9a523ed3268688e90ee18f1ecd93f4fb72a8f056`. The harness verifies the immutable public manifest against all registered IDs and canonical hashes before starting. + +It adopts a unique matching scratch/balanced lesson, persists submission intent before POST, and polls the same job ID. Ambiguous or missing jobs after an uncertain POST stop the observer rather than authorizing a duplicate submission. Checked READY and NEEDS_MORE drafts are evaluated across all canonical primary and supplied alternate prompts. Execution completion and answer accuracy are separate counters. Failed, cancelled, expired or unchecked lessons remain explicit failures, not dropped denominators. + +The inference audit checkpoints raw response hashes. Resume validates those hashes and does not repeat checkpointed chat calls. Old unversioned evidence is never overwritten. Unknown patches and missing restoration journals block cleanup; only the exact owned draft can be unloaded. Run this observer exclusively on a dedicated experiment runtime: checks do not provide a distributed reservation against another operator's API requests. + +## Tests + +Node 24, no third-party dependencies: + +```sh +node --test scripts/year3-dart100/test/ainize-lifecycle.test.js +``` + +Nine regression tests cover dataset binding, uncertain submissions, job identity collisions, stack ownership, interrupted audits, historical evidence protection, tampering, and canonical denominators. They use fake CLI/model responses and do not establish real GPU training success. + +## Existing deployment + +This is not a fresh-machine bootstrap. Current paths are `/mnt/newdata/gov/kpi`, Ainize CLI `/opt/ainize/ainize-cli/dist/bin.js`, and API `http://localhost:3410`. Existing operator credentials, registered datasets, immutable HF publication evidence, and the actual PLE model/trainer are prerequisites. Do not copy secret homes, API tokens or `.env` into this repository. + +Place these four JavaScript modules in a frozen source directory under `kpi/evidence//source`, preserve their SHA256 manifest, and execute within the configured Ainize Docker container. The deployment wrapper also snapshots Ainize, serving and trainer image IDs and Docker CPU/GPU/memory limits. + +```sh +docker exec -e RUN_ID=ainize_lifecycle100_20260911 \ + ain-cert-ainize-node-1 \ + flock --nonblock --no-fork /mnt/newdata/gov/kpi/evidence/.ainize-lifecycle.lock \ + node /mnt/newdata/gov/kpi/evidence/ainize_lifecycle100_20260911/source/ainize-lifecycle.js +``` + +Reuse the same RUN_ID and unchanged snapshot when resuming. The shared flock prevents concurrent copies of this harness. An observer failure does not cancel the server-side lesson. Inspect the saved intent, job ID, raw responses and runtime queue before restarting; never restart the model simply because a client timed out. + +`progress.json` distinguishes `complete` (all datasets visited), `executionComplete` (all 100 datasets have complete inference observations), and `allAnswersCorrect`. Exit zero means execution coverage only, not all answers correct or overall certification success. This code does not publish drafts, waive consent, bypass PII checks or award blockchain incentives. + +## Observed limits + +The initial live run adopts job `6314e86b-9ba2-4bc3-8a21-e3294663fdd7` without retraining. Its isolated 16-call audit reports 5/8 primary and 1/8 alternate answers correct; it is not a quality pass. A second registered dataset is training under a different job ID. This is not evidence of 100 completed lessons. The running snapshot predates the additional completed-audit resume hash check and unique CLI-error filenames in this source; those additions are regression-tested, not retroactively attributed to that run. diff --git a/scripts/year3-dart100/ainize-inference-audit.js b/scripts/year3-dart100/ainize-inference-audit.js new file mode 100644 index 0000000..b6de48b --- /dev/null +++ b/scripts/year3-dart100/ainize-inference-audit.js @@ -0,0 +1,180 @@ +const fs = require('fs'); +const path = require('path'); +const assert = require('assert/strict'); +const { execFile } = require('child_process'); +const { promisify } = require('util'); +const execute = promisify(execFile); +const { hitOf } = require('./m5-judge'); +const { sha256, assertOwnedStack } = require('./ainize-lifecycle-state'); + +function saveJson(filename, value) { + const temporary = `${filename}.tmp`; + const descriptor = fs.openSync(temporary, 'w'); + try { + fs.writeFileSync(descriptor, JSON.stringify(value, null, 2) + '\n'); + fs.fsyncSync(descriptor); + } finally { fs.closeSync(descriptor); } + fs.renameSync(temporary, filename); + const directory = fs.openSync(path.dirname(filename), 'r'); + try { fs.fsyncSync(directory); } finally { fs.closeSync(directory); } +} + +function makeCli(output) { + return async (args, name) => { + const filename = path.join(output, `${name}.json`); + assert.ok(!fs.existsSync(filename), `raw response already exists: ${name}`); + try { + const result = await execute(process.execPath, ['/opt/ainize/ainize-cli/dist/bin.js', ...args, '--json'], { timeout: 300000, maxBuffer: 8 * 1024 * 1024 }); + fs.writeFileSync(filename, result.stdout, { flag: 'wx' }); + return JSON.parse(result.stdout); + } catch (error) { + fs.writeFileSync(`${filename}.error-${Date.now()}-${process.pid}.json`, JSON.stringify({ message: error.message, stdout: error.stdout, stderr: error.stderr }), { flag: 'wx' }); + throw error; + } + }; +} + +async function runtimeStatus() { + const response = await fetch('http://localhost:3410/api/info', { signal: AbortSignal.timeout(20000) }); + assert.ok(response.ok, 'runtime info HTTP error'); + return (await response.json()).runtime; +} + +function assertIdle(runtime) { + assert.equal(runtime.available, true, 'runtime unavailable'); + assert.equal(runtime.queue?.running, null, 'runtime busy; leave current operation intact'); + assert.equal(runtime.queue.waiting, 0, 'runtime queue is not empty'); + assert.equal(runtime.queue.lock, null, 'runtime is locked'); +} + +function score(inference, fact, index, kind, rawFile) { + const valid = answer => typeof answer?.content === 'string' && answer.content.trim().length > 0 && + Number.isInteger(answer.usage?.completion_tokens) && answer.usage.completion_tokens > 0 && answer.usage.completion_tokens <= 256 && + answer.finish_reason === 'stop' && !answer.truncated; + return { index, kind, prompt: kind === 'primary' ? fact.prompt : fact.alt_prompt, expected: fact.answer, model: inference.model, + baseValid: valid(inference.base), patchedValid: valid(inference.patched), + baseHit: valid(inference.base) && hitOf(inference.base.content, fact.answer), + patchedHit: valid(inference.patched) && hitOf(inference.patched.content, fact.answer), + nodeBenchmarkHit: inference.benchmark_hit, baseTokens: inference.base?.usage?.completion_tokens, + patchedTokens: inference.patched?.usage?.completion_tokens, rawFile }; +} + +async function runAudit({ runId, jobId, registrationRun = 'ainize_datasets100_20260911', root = '/mnt/newdata/gov/kpi', output, cli, getRuntime = runtimeStatus }) { + for (const value of [runId, jobId, registrationRun]) assert.ok(value && /^[A-Za-z0-9_-]+$/.test(value), 'valid run and job IDs required'); + output ||= path.join(root, 'evidence', runId); + fs.mkdirSync(output, { recursive: true }); + const progressFile = path.join(output, 'progress.json'); + let progress = { version: 2, runId, jobId, registrationRun, startedAt: new Date().toISOString(), complete: false, attempts: 0, samples: [], failures: [] }; + if (fs.existsSync(progressFile)) { + progress = JSON.parse(fs.readFileSync(progressFile)); + assert.equal(progress.version, 2, 'legacy evidence is immutable; use a new audit directory'); + assert.equal(progress.runId, runId); + assert.equal(progress.jobId, jobId); + assert.equal(progress.registrationRun, registrationRun); + } + progress.attempts++; + const attempt = `attempt-${progress.attempts}`; + fs.mkdirSync(path.join(output, attempt)); + cli ||= makeCli(output); + const fresh = (args, name) => cli(args, `${attempt}/${name}`); + const save = () => saveJson(progressFile, progress); + save(); + try { + const registration = JSON.parse(fs.readFileSync(path.join(root, 'evidence', registrationRun, 'progress.json'))); + const { job } = await fresh(['teach', 'status', jobId], 'job-before'); + assert.equal(job.id, jobId); + assert.ok(['READY', 'NEEDS_MORE'].includes(job.status), 'no checked draft to audit'); + assert.equal(job.checks.executed, true); + assert.ok(job.draft_id && /^[a-f0-9]{64}$/.test(job.result?.sha256)); + assert.equal(job.mode, 'scratch'); + assert.deepEqual(job.context_patch_ids, []); + const dataset = registration.datasets.find(entry => entry.datasetId === job.dataset.id); + assert.ok(dataset?.ok); + assert.equal(dataset.canonicalSha256, job.dataset.sha256); + assert.equal(dataset.rows, job.dataset.rows); + const downloaded = await fresh(['teach', 'dataset', 'get', dataset.datasetId], 'dataset'); + assert.equal(downloaded.dataset.id, dataset.datasetId); + assert.equal(downloaded.dataset.sha256, dataset.canonicalSha256); + assert.equal(downloaded.dataset.rows, dataset.rows); + assert.equal(downloaded.dataset.revision, job.dataset.revision); + assert.ok(downloaded.dataset.job_ids.includes(jobId)); + const canonical = fs.readFileSync(path.join(root, 'evidence', registrationRun, `${dataset.lessonId}-canonical.jsonl`)); + assert.equal(sha256(canonical), dataset.canonicalSha256); + const facts = canonical.toString('utf8').trim().split('\n').map(line => JSON.parse(line)); + assert.equal(facts.length, dataset.rows); + const content = fact => ({ prompt: fact.prompt, answer: fact.answer, alt_prompt: fact.alt_prompt || null }); + for (const fact of job.facts) assert.ok(facts.some(original => JSON.stringify(content(original)) === JSON.stringify(content(fact))), 'trained fact differs from canonical dataset'); + const patch = { id: job.draft_id, sha256: job.result.sha256 }; + if (progress.patch) assert.deepEqual(progress.patch, patch); + progress.dataset = { id: dataset.datasetId, sha256: dataset.canonicalSha256, rows: dataset.rows, lessonId: dataset.lessonId }; + progress.patch = patch; + progress.publishStatus = job.publish_status; + progress.scope = 'All canonical dataset rows through Ainize compare inference; answer accuracy is separate from execution, public listing and distinct model count'; + const cleanStack = async name => { + const runtime = await getRuntime(); + saveJson(path.join(output, attempt, `${name}-runtime.json`), runtime); + assertIdle(runtime); + const stack = await fresh(['patch', 'stack'], `${name}-before`); + assertOwnedStack(stack, patch); + if (stack.length) await fresh(['patch', 'remove', patch.id], `${name}-remove`); + assert.deepEqual(await fresh(['patch', 'stack'], `${name}-after`), []); + }; + await cleanStack('baseline'); + const samples = new Map(progress.samples.map(sample => [sample.rawFile, sample])); + assert.equal(samples.size, progress.samples.length, 'duplicate saved sample'); + progress.complete = false; + save(); + let expectedSamples = 0; + for (const [index, fact] of facts.entries()) { + for (const [kind, prompt] of [['primary', fact.prompt], ['heldout', fact.alt_prompt]]) { + if (!prompt) continue; + expectedSamples++; + const name = `fact-${index}-${kind}`; + const rawFile = `${name}.json`; + const filename = path.join(output, rawFile); + let inference; + if (fs.existsSync(filename)) { + const bytes = fs.readFileSync(filename); + assert.equal(samples.get(rawFile)?.sha256, sha256(bytes), 'uncheckpointed or changed raw inference; inspect before retry'); + inference = JSON.parse(bytes); + } else { + assert.ok(!samples.has(rawFile), 'checkpointed raw inference is missing'); + const stack = await fresh(['patch', 'stack'], `${name}-stack`); + assertOwnedStack(stack, patch); + inference = await cli(['chat', patch.id, prompt, '--mode', 'compare', '--max-tokens', '256'], name); + } + assert.equal(inference.patch_id, patch.id); + assert.equal(inference.mode, 'compare'); + assert.deepEqual(inference.dirty, []); + samples.set(rawFile, { ...score(inference, fact, index, kind, rawFile), sha256: sha256(fs.readFileSync(filename)) }); + progress.samples = [...samples.values()]; + save(); + } + } + assert.equal(progress.samples.length, expectedSamples, 'unexpected saved samples'); + await cleanStack('finished'); + const final = await fresh(['teach', 'status', jobId], 'job-after'); + assert.equal(final.job.result.sha256, patch.sha256); + assert.equal(final.job.dataset.sha256, dataset.canonicalSha256); + progress.complete = true; + progress.finishedAt = new Date().toISOString(); + progress.primary = { total: facts.length, hits: progress.samples.filter(sample => sample.kind === 'primary' && sample.patchedHit).length }; + progress.heldout = { total: facts.filter(fact => fact.alt_prompt).length, hits: progress.samples.filter(sample => sample.kind === 'heldout' && sample.patchedHit).length }; + progress.pass = progress.primary.hits === progress.primary.total && progress.heldout.hits === progress.heldout.total; + progress.error = null; + save(); + return progress; + } catch (error) { + progress.complete = false; + progress.error = error.message; + progress.failures.push({ at: new Date().toISOString(), attempt, message: error.message }); + save(); + throw error; + } +} + +if (require.main === module) runAudit({ runId: process.env.RUN_ID, jobId: process.env.AINIZE_JOB_ID, registrationRun: process.env.AINIZE_REGISTRATION_RUN }) + .then(progress => { console.log(JSON.stringify({ runId: progress.runId, dataset: progress.dataset, primary: progress.primary, heldout: progress.heldout, pass: progress.pass })); process.exitCode = progress.pass ? 0 : 1; }) + .catch(error => { console.error(error.message); process.exitCode = 1; }); + +module.exports = { runAudit, saveJson, makeCli, score, runtimeStatus, assertIdle }; diff --git a/scripts/year3-dart100/ainize-lifecycle-state.js b/scripts/year3-dart100/ainize-lifecycle-state.js new file mode 100644 index 0000000..3616ed0 --- /dev/null +++ b/scripts/year3-dart100/ainize-lifecycle-state.js @@ -0,0 +1,87 @@ +const assert = require('assert/strict'); +const crypto = require('crypto'); + +const sha256 = bytes => crypto.createHash('sha256').update(bytes).digest('hex'); +const activeStatuses = new Set(['QUEUED', 'PREFLIGHT', 'LOADING', 'TRAINING', 'EXPORTED', 'CHECKING']); +const terminalStatuses = new Set(['READY', 'NEEDS_MORE', 'FAILED', 'CANCELLED', 'PENDING_REVIEW', 'REJECTED', 'ANNOUNCED', 'EXPIRED']); + +function entriesFrom(registrationBytes, manifest) { + const registration = JSON.parse(registrationBytes); + assert.equal(manifest.registrationSha256, sha256(registrationBytes), 'registration hash changed'); + assert.equal(registration.datasets.length, 100); + assert.equal(manifest.datasets.length, 100); + for (const key of ['datasetId', 'lessonId', 'canonicalSha256']) { + assert.equal(new Set(registration.datasets.map(dataset => dataset[key])).size, 100, `duplicate ${key}`); + } + assert.equal(new Set(manifest.datasets.map(dataset => dataset.config)).size, 100, 'duplicate HF config'); + return registration.datasets.map(dataset => { + assert.equal(dataset.ok, true); + assert.ok(Object.keys(dataset.checks).length > 0 && Object.values(dataset.checks).every(value => value === true)); + assert.ok(/^[A-Za-z0-9_-]+$/.test(dataset.lessonId)); + assert.ok(/^[A-Za-z0-9_-]+$/.test(dataset.datasetId)); + assert.ok(/^[a-f0-9]{64}$/.test(dataset.canonicalSha256)); + assert.ok(Number.isInteger(dataset.rows) && dataset.rows > 0); + const published = manifest.datasets.find(entry => entry.config === dataset.lessonId); + assert.ok(published, 'missing HF config'); + assert.equal(published.ainizeDatasetId, dataset.datasetId); + assert.equal(published.sha256, dataset.canonicalSha256); + assert.equal(published.rows, dataset.rows); + return { lessonId: dataset.lessonId, datasetId: dataset.datasetId, sha256: dataset.canonicalSha256, rows: dataset.rows }; + }); +} + +function validateJob(entry, job) { + assert.ok(job?.id, 'missing job'); + assert.equal(job.dataset?.id, entry.datasetId, 'job dataset ID mismatch'); + assert.equal(job.dataset.sha256, entry.sha256, 'job dataset hash mismatch'); + assert.equal(job.dataset.rows, entry.rows, 'job dataset rows mismatch'); + assert.equal(job.mode, 'scratch', 'only independent scratch lessons are supported'); + assert.deepEqual(job.context_patch_ids, []); + assert.equal(job.training?.effort, 'balanced'); + assert.ok(activeStatuses.has(job.status) || terminalStatuses.has(job.status), `unknown job status ${job.status}`); + return job; +} + +function selectJob(entry, jobs) { + assert.ok(Array.isArray(jobs)); + const sameName = jobs.filter(job => job.name === entry.name); + assert.ok(sameName.length <= 1, 'ambiguous job name; inspect jobs without resubmitting'); + if (sameName.length) validateJob(entry, sameName[0]); + if (entry.jobId) { + const existing = jobs.filter(job => job.id === entry.jobId); + assert.equal(existing.length, 1, 'saved job is missing; do not resubmit'); + return validateJob(entry, existing[0]); + } + if (sameName.length) return sameName[0]; + assert.ok(!entry.submissionIntent, 'submission outcome unknown; inspect jobs without resubmitting'); + const sameDataset = jobs.filter(job => job.dataset?.id === entry.datasetId); + assert.ok(sameDataset.length <= 1, 'multiple existing lessons for this dataset; explicit reconciliation required'); + return sameDataset.length ? validateJob(entry, sameDataset[0]) : null; +} + +function assertOwnedStack(stack, patch) { + assert.ok(Array.isArray(stack), 'stack response is not an array'); + assert.ok(stack.length <= 1, 'other patches are loaded; not removing them'); + for (const layer of stack) { + assert.equal(layer.patch_id, patch.id, 'unowned patch loaded; not removing it'); + assert.equal(layer.sha256, patch.sha256, 'loaded patch hash changed'); + assert.equal(layer.journal, true, 'missing restoration journal'); + } +} + +function summarize(entries) { + const audited = entries.filter(entry => entry.audit?.complete); + return { + target: entries.length, + jobs: entries.filter(entry => entry.jobId).length, + ready: entries.filter(entry => entry.status === 'READY').length, + needsMore: entries.filter(entry => entry.status === 'NEEDS_MORE').length, + terminal: entries.filter(entry => terminalStatuses.has(entry.status)).length, + inferenceComplete: audited.length, + allAnswersCorrect: audited.filter(entry => entry.audit.pass).length, + primary: { hits: audited.reduce((total, entry) => total + entry.audit.primary.hits, 0), total: audited.reduce((total, entry) => total + entry.audit.primary.total, 0) }, + heldout: { hits: audited.reduce((total, entry) => total + entry.audit.heldout.hits, 0), total: audited.reduce((total, entry) => total + entry.audit.heldout.total, 0) } + }; +} + +module.exports = { sha256, activeStatuses, terminalStatuses, entriesFrom, validateJob, selectJob, assertOwnedStack, summarize }; diff --git a/scripts/year3-dart100/ainize-lifecycle.js b/scripts/year3-dart100/ainize-lifecycle.js new file mode 100644 index 0000000..531bf1e --- /dev/null +++ b/scripts/year3-dart100/ainize-lifecycle.js @@ -0,0 +1,124 @@ +const fs = require('fs'); +const path = require('path'); +const assert = require('assert/strict'); +const { entriesFrom, sha256, selectJob, validateJob, activeStatuses, summarize } = require('./ainize-lifecycle-state'); +const { runAudit, makeCli, saveJson, runtimeStatus, assertIdle } = require('./ainize-inference-audit'); + +async function main() { + const root = '/mnt/newdata/gov/kpi'; + const runId = process.env.RUN_ID; + assert.ok(runId && /^[A-Za-z0-9_-]{1,40}$/.test(runId), 'RUN_ID must be 1-40 safe characters'); + const output = path.join(root, 'evidence', runId); + const registrationRun = process.env.AINIZE_REGISTRATION_RUN || 'ainize_datasets100_20260911'; + const publicationRun = process.env.HF_PUBLICATION_RUN || 'hf_datasets_publish_r2_20260911'; + for (const value of [registrationRun, publicationRun]) assert.ok(/^[A-Za-z0-9_-]+$/.test(value)); + const registrationBytes = fs.readFileSync(path.join(root, 'evidence', registrationRun, 'progress.json')); + const publication = JSON.parse(fs.readFileSync(path.join(root, 'evidence', publicationRun, 'publication.json'))); + assert.equal(publication.pass, true); + assert.equal(publication.verifiedFiles, 202); + assert.ok(/^[a-f0-9]{40}$/.test(publication.commit)); + assert.ok(/^[A-Za-z0-9_-]+\/[A-Za-z0-9_.-]+$/.test(publication.repoId)); + const manifestBytes = fs.readFileSync(path.join(root, 'evidence', publicationRun, 'download', 'manifest.json')); + const manifest = JSON.parse(manifestBytes); + const publishedResponse = await fetch(`https://huggingface.co/datasets/${publication.repoId}/resolve/${publication.commit}/manifest.json`, { signal: AbortSignal.timeout(60000) }); + assert.ok(publishedResponse.ok, 'public HF manifest unavailable'); + assert.equal(sha256(Buffer.from(await publishedResponse.arrayBuffer())), sha256(manifestBytes), 'public manifest mismatch'); + const entries = entriesFrom(registrationBytes, manifest); + for (const entry of entries) { + assert.equal(sha256(fs.readFileSync(path.join(root, 'evidence', registrationRun, `${entry.lessonId}-canonical.jsonl`))), entry.sha256); + } + const identity = { runId, registrationRun, registrationSha256: sha256(registrationBytes), repoId: publication.repoId, revision: publication.commit, manifestSha256: sha256(manifestBytes) }; + const filename = path.join(output, 'progress.json'); + let state = { version: 1, identity, startedAt: new Date().toISOString(), attempts: 0, complete: false, entries: entries.map((entry, index) => ({ ...entry, name: `${runId}-${String(index + 1).padStart(3, '0')}`, status: 'PENDING' })) }; + if (fs.existsSync(filename)) { + state = JSON.parse(fs.readFileSync(filename)); + assert.equal(state.version, 1); + assert.deepEqual(state.identity, identity, 'resume inputs changed'); + assert.deepEqual(state.entries.map(({ lessonId, datasetId, sha256: hash, rows }) => ({ lessonId, datasetId, sha256: hash, rows })), entries); + } + state.attempts++; + const attempt = `attempt-${state.attempts}`; + fs.mkdirSync(path.join(output, attempt)); + let command = 0; + const cli = makeCli(output); + const call = (args, name) => cli(args, `${attempt}/${String(++command).padStart(5, '0')}-${name}`); + const save = () => { + state.updatedAt = new Date().toISOString(); + state.summary = summarize(state.entries); + saveJson(filename, state); + }; + save(); + try { + for (const entry of state.entries) { + if (entry.done) { + if (entry.audit) { + assert.equal(entry.audit.file, `${entry.lessonId}/progress.json`); + const auditFolder = path.join(output, entry.lessonId); + const auditBytes = fs.readFileSync(path.join(auditFolder, 'progress.json')); + assert.equal(sha256(auditBytes), entry.audit.sha256, 'completed audit changed'); + const audit = JSON.parse(auditBytes); + assert.equal(audit.complete, true); + assert.equal(audit.jobId, entry.jobId); + assert.equal(audit.dataset.sha256, entry.sha256); + for (const sample of audit.samples) { + assert.ok(/^fact-[0-9]+-(primary|heldout)\.json$/.test(sample.rawFile)); + assert.equal(sha256(fs.readFileSync(path.join(auditFolder, sample.rawFile))), sample.sha256, 'completed raw inference changed'); + } + } + continue; + } + const { items } = await call(['teach', 'jobs'], 'jobs'); + assert.ok(Array.isArray(items) && items.length < 500, 'job list may be truncated'); + let job = selectJob(entry, items); + const foreign = items.filter(item => activeStatuses.has(item.status) && item.id !== job?.id); + assert.equal(foreign.length, 0, 'another lesson is active; do not queue duplicate work'); + if (!job) { + assertIdle(await runtimeStatus()); + assert.deepEqual(await call(['patch', 'stack'], 'stack-before-submit'), [], 'scratch training requires an empty stack'); + entry.submissionIntent = { at: new Date().toISOString(), datasetId: entry.datasetId, sha256: entry.sha256, name: entry.name, effort: 'balanced' }; + save(); + const response = await call(['teach', 'train', entry.datasetId, '--name', entry.name, '--effort', 'balanced'], 'submitted'); + job = validateJob(entry, response.job); + entry.submitted = true; + } else { entry.adopted = !entry.submitted; } + entry.jobId = job.id; + entry.status = job.status; + save(); + while (activeStatuses.has(job.status)) { + await new Promise(resolve => setTimeout(resolve, 30000)); + job = validateJob(entry, (await call(['teach', 'status', entry.jobId], 'status')).job); + entry.status = job.status; + entry.progress = job.progress; + entry.blocked = job.blocked; + save(); + console.log(JSON.stringify({ at: state.updatedAt, lessonId: entry.lessonId, jobId: job.id, status: job.status, progress: job.progress, blocked: job.blocked })); + } + job = validateJob(entry, (await call(['teach', 'status', entry.jobId], 'terminal')).job); + entry.status = job.status; + entry.terminal = { draftId: job.draft_id, patchSha256: job.result?.sha256, checks: job.checks, publishStatus: job.publish_status, error: job.error }; + save(); + if (['READY', 'NEEDS_MORE'].includes(job.status) && job.checks?.executed && job.draft_id) { + const auditFolder = path.join(output, entry.lessonId); + const audit = await runAudit({ runId: `${runId}-${entry.lessonId}`, jobId: job.id, root, registrationRun, output: auditFolder }); + entry.audit = { complete: audit.complete, pass: audit.pass, primary: audit.primary, heldout: audit.heldout, file: `${entry.lessonId}/progress.json`, sha256: sha256(fs.readFileSync(path.join(auditFolder, 'progress.json'))) }; + } else { entry.inferenceUnavailable = `terminal status ${job.status}; checked draft ${Boolean(job.checks?.executed && job.draft_id)}`; } + entry.done = true; + save(); + console.log(JSON.stringify({ lessonId: entry.lessonId, summary: state.summary })); + } + state.complete = true; + state.finishedAt = new Date().toISOString(); + state.executionComplete = state.entries.every(entry => entry.audit?.complete); + state.allAnswersCorrect = state.entries.every(entry => entry.audit?.pass); + state.scope = '100 dataset lifecycle observations on one base model; neither 100 distinct models nor public Ainize listing nor HF model deployment'; + state.error = null; + save(); + process.exitCode = state.executionComplete ? 0 : 1; + } catch (error) { + state.error = error.message; + save(); + throw error; + } +} + +if (require.main === module) main().catch(error => { console.error(error.message); process.exitCode = 1; }); diff --git a/scripts/year3-dart100/m5-judge.js b/scripts/year3-dart100/m5-judge.js new file mode 100644 index 0000000..b2ca1b7 --- /dev/null +++ b/scripts/year3-dart100/m5-judge.js @@ -0,0 +1,34 @@ +// M5 정답 판정 (m5-ainize.js · test/hitOf.test.js 공용) +// 규칙: 공백·마크다운 기호 제거, 천단위 콤마 제거 후 정답이 응답 안에 '경계가 분리된 토큰' 으로 나타나야 한다. +// 경계 위반 = +// (a) 인접 문자가 영숫자 ('35203', 'abc352') +// (b) 숫자 구분자(. , -)가 숫자와 이어져 더 긴 수를 이루는 경우 ('352.7', '0.352', '1,352,000', '12-352', '-352') +// (c) 정답 가장자리가 한글이고 인접 문자도 한글인데 조사·서술어가 아닌 경우 ('김준' vs '김준호', '홍김준') +// — 한국어는 조사가 붙어 쓰이므로('조원국입니다', '대표이사는김준') 조사·계사는 경계로 인정한다. +// 응답이 JSON 오브젝트(서버 에러 본문)면 항상 불일치. +const norm = x => String(x ?? '').replace(/\s+/g, '').replace(/[*_`]/g, '').replace(/(\d),(?=\d{3})/g, '$1'); +const ALNUM = /[A-Za-z0-9]/, HANGUL = /[가-힣]/; +const PARTICLE_AFTER = /^(입니다|이다|임|은|는|이|가|을|를|의|과|와|로|으로|에서|에|도|이며|이고|이었|였|으로서|로서|입)/; +const PARTICLE_BEFORE = /[은는이가의과와로에서도,:(]$/; +function hitOf(content, answer) { + if (typeof content !== 'string' || content.trim().startsWith('{')) return false; + const c = norm(content), a = norm(answer); + if (!a) return false; + const startHangul = HANGUL.test(a[0]), endHangul = HANGUL.test(a[a.length - 1]); + let from = 0; + while (true) { + const i = c.indexOf(a, from); + if (i < 0) return false; + const before = i > 0 ? c[i - 1] : '', before2 = i > 1 ? c[i - 2] : ''; + const after = c[i + a.length] || '', after2 = c[i + a.length + 1] || ''; + const badBefore = ALNUM.test(before) + || (/[.,\-]/.test(before) && (/[0-9]/.test(before2) || (before === '-' && /^[0-9]/.test(a)))) + || (HANGUL.test(before) && startHangul && !PARTICLE_BEFORE.test(before)); + const badAfter = ALNUM.test(after) + || (/[.,\-]/.test(after) && /[0-9]/.test(after2)) + || (HANGUL.test(after) && endHangul && !PARTICLE_AFTER.test(c.slice(i + a.length, i + a.length + 4))); + if (!badBefore && !badAfter) return true; + from = i + 1; + } +} +module.exports = { norm, hitOf }; diff --git a/scripts/year3-dart100/package.json b/scripts/year3-dart100/package.json new file mode 100644 index 0000000..c18e009 --- /dev/null +++ b/scripts/year3-dart100/package.json @@ -0,0 +1,11 @@ +{ + "name": "ainize-dart100-evidence", + "private": true, + "type": "commonjs", + "scripts": { + "test": "node --test test/ainize-lifecycle.test.js" + }, + "engines": { + "node": ">=24" + } +} diff --git a/scripts/year3-dart100/test/ainize-lifecycle.test.js b/scripts/year3-dart100/test/ainize-lifecycle.test.js new file mode 100644 index 0000000..5026c2f --- /dev/null +++ b/scripts/year3-dart100/test/ainize-lifecycle.test.js @@ -0,0 +1,136 @@ +const test = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const { entriesFrom, sha256, selectJob, assertOwnedStack, summarize } = require('../ainize-lifecycle-state'); +const { runAudit, saveJson, assertIdle } = require('../ainize-inference-audit'); + +function registrationFixture() { + const datasets = Array.from({ length: 100 }, (_, index) => ({ ok: true, datasetId: `dataset-${index}`, lessonId: `dart-${index}`, canonicalSha256: sha256(String(index)), rows: 8, checks: { preserved: true } })); + const bytes = Buffer.from(JSON.stringify({ datasets })); + const manifest = { registrationSha256: sha256(bytes), datasets: datasets.map(dataset => ({ config: dataset.lessonId, ainizeDatasetId: dataset.datasetId, sha256: dataset.canonicalSha256, rows: dataset.rows })) }; + return { bytes, manifest }; +} + +const entry = { datasetId: 'dataset-1', name: 'run-001', sha256: sha256('rows'), rows: 2 }; +const job = { id: 'job-1', name: entry.name, dataset: { id: entry.datasetId, sha256: entry.sha256, rows: 2 }, mode: 'scratch', context_patch_ids: [], training: { effort: 'balanced' }, status: 'TRAINING' }; + +test('exactly 100 unique registered configurations bind to the published hashes', () => { + const fixture = registrationFixture(); + assert.equal(entriesFrom(fixture.bytes, fixture.manifest).length, 100); + fixture.manifest.datasets[0].sha256 = sha256('changed'); + assert.throws(() => entriesFrom(fixture.bytes, fixture.manifest), /strictly equal/); + const duplicate = registrationFixture(); + duplicate.manifest.datasets[1] = duplicate.manifest.datasets[0]; + assert.throws(() => entriesFrom(duplicate.bytes, duplicate.manifest), /duplicate HF config/); +}); + +test('existing jobs are adopted, including after an uncertain submission', () => { + assert.equal(selectJob(entry, []), null); + assert.equal(selectJob(entry, [job]), job); + assert.equal(selectJob({ ...entry, submissionIntent: {} }, [job]), job); + assert.equal(selectJob({ ...entry, jobId: job.id }, [job]), job); + assert.throws(() => selectJob({ ...entry, submissionIntent: {} }, []), /outcome unknown/); + assert.throws(() => selectJob({ ...entry, jobId: job.id }, []), /saved job is missing/); +}); + +test('hash mismatches, ambiguous jobs and unknown statuses never authorize submission', () => { + assert.throws(() => selectJob(entry, [{ ...job, dataset: { ...job.dataset, sha256: sha256('changed') } }]), /hash mismatch/); + assert.throws(() => selectJob(entry, [job, { ...job, id: 'job-2' }]), /ambiguous/); + assert.throws(() => selectJob(entry, [{ ...job, status: 'NEW_UNKNOWN_STATE' }]), /unknown job status/); +}); + +test('only the owned hash with a journal can be unloaded', () => { + const patch = { id: 'own', sha256: sha256('patch') }; + assertOwnedStack([], patch); + assertOwnedStack([{ patch_id: patch.id, sha256: patch.sha256, journal: true }], patch); + assert.throws(() => assertOwnedStack([{ patch_id: 'someone-else' }], patch), /unowned/); + assert.throws(() => assertOwnedStack([{ patch_id: patch.id, sha256: patch.sha256, journal: false }], patch), /journal/); + assert.throws(() => assertIdle({ available: true, queue: { running: { label: 'other' }, waiting: 0, lock: null } }), /busy/); +}); + +function auditFixture(context) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'ainize-audit-')); + context.after(() => fs.rmSync(root, { recursive: true, force: true })); + const registrationFolder = path.join(root, 'evidence', 'registration'); + const output = path.join(root, 'evidence', 'audit'); + fs.mkdirSync(registrationFolder, { recursive: true }); + fs.mkdirSync(output); + const facts = [{ prompt: '대표자는?', answer: '조원국', alt_prompt: '대표자 이름?' }, { prompt: '종목코드는?', answer: '001740' }]; + const canonical = facts.map(fact => JSON.stringify(fact)).join('\n') + '\n'; + const dataset = { id: 'dataset-1', sha256: sha256(canonical), rows: 2, revision: 1, job_ids: ['job-1'] }; + const checkedJob = { ...job, status: 'READY', dataset, draft_id: 'patch-1', result: { sha256: sha256('patch') }, facts, checks: { executed: true }, publish_status: 'none' }; + saveJson(path.join(registrationFolder, 'progress.json'), { datasets: [{ datasetId: dataset.id, canonicalSha256: dataset.sha256, lessonId: 'dart-1', rows: 2, ok: true }] }); + fs.writeFileSync(path.join(registrationFolder, 'dart-1-canonical.jsonl'), canonical); + const state = { chats: 0, removes: 0, stack: [], failAt: null, runtime: { available: true, queue: { running: null, waiting: 0, lock: null } } }; + const cli = async (args, name) => { + let result; + if (args[0] === 'teach' && args[1] === 'status') result = { job: checkedJob }; + else if (args[0] === 'teach' && args[1] === 'dataset') result = { dataset }; + else if (args[0] === 'patch' && args[1] === 'stack') result = state.stack; + else if (args[0] === 'patch' && args[1] === 'remove') { state.stack = []; state.removes++; result = { applied: [] }; } + else if (args[0] === 'chat') { + state.chats++; + if (state.chats === state.failAt) throw new Error('simulated interrupted observation'); + const answer = { content: '조원국', usage: { completion_tokens: 3 }, finish_reason: 'stop', truncated: null }; + result = { patch_id: checkedJob.draft_id, mode: 'compare', dirty: [], base: answer, patched: answer, model: 'fixture' }; + state.stack = [{ patch_id: checkedJob.draft_id, sha256: checkedJob.result.sha256, journal: true }]; + } else throw new Error(`unexpected command ${args}`); + fs.writeFileSync(path.join(output, `${name}.json`), JSON.stringify(result), { flag: 'wx' }); + return result; + }; + return { state, checkedJob, output, options: { root, output, runId: 'audit', jobId: 'job-1', registrationRun: 'registration', cli, getRuntime: async () => state.runtime } }; +} + +test('interrupted audit resumes saved samples without duplicate chat and records accuracy separately', async context => { + const fixture = auditFixture(context); + fixture.state.failAt = 2; + await assert.rejects(runAudit(fixture.options), /interrupted/); + assert.equal(JSON.parse(fs.readFileSync(path.join(fixture.output, 'progress.json'))).samples.length, 1); + const result = await runAudit(fixture.options); + assert.equal(fixture.state.chats, 4); + assert.equal(result.complete, true); + assert.equal(result.pass, false); + assert.deepEqual(result.primary, { hits: 1, total: 2 }); + assert.deepEqual(result.heldout, { hits: 1, total: 1 }); + assert.deepEqual(fixture.state.stack, []); + await runAudit(fixture.options); + assert.equal(fixture.state.chats, 4); + const summary = summarize([{ jobId: 'job-1', status: 'READY', audit: result }]); + assert.equal(summary.inferenceComplete, 1); + assert.equal(summary.allAnswersCorrect, 0); +}); + +test('legacy evidence cannot be overwritten by a failed retry', async context => { + const fixture = auditFixture(context); + const filename = path.join(fixture.output, 'progress.json'); + const original = JSON.stringify({ runId: 'legacy', complete: true }); + fs.writeFileSync(filename, original); + await assert.rejects(runAudit(fixture.options), /legacy evidence/); + assert.equal(fs.readFileSync(filename, 'utf8'), original); +}); + +test('an unowned stack halts before any inference or unload', async context => { + const fixture = auditFixture(context); + fixture.state.stack = [{ patch_id: 'another-user' }]; + await assert.rejects(runAudit(fixture.options), /unowned/); + assert.equal(fixture.state.chats, 0); + assert.equal(fixture.state.removes, 0); +}); + +test('tampered raw responses fail even when the saved report claimed success', async context => { + const fixture = auditFixture(context); + await runAudit(fixture.options); + fs.appendFileSync(path.join(fixture.output, 'fact-0-primary.json'), ' '); + await assert.rejects(runAudit(fixture.options), /changed raw inference/); + assert.equal(fixture.state.chats, 3); +}); + +test('canonical rows dropped during training remain in the inference denominator', async context => { + const fixture = auditFixture(context); + fixture.checkedJob.facts = fixture.checkedJob.facts.slice(0, 1); + const result = await runAudit(fixture.options); + assert.equal(result.primary.total, 2); + assert.equal(result.samples.length, 3); +}); From a85bfda85c13d3b75a1d6a3bd34983a0f9dfc520 Mon Sep 17 00:00:00 2001 From: Haechan Date: Fri, 11 Sep 2026 07:21:34 +0000 Subject: [PATCH 2/5] fix(dart100): require integration rather than HF publication --- scripts/year3-dart100/README.md | 8 ++++---- .../year3-dart100/ainize-lifecycle-state.js | 20 +++++++++++-------- scripts/year3-dart100/ainize-lifecycle.js | 19 ++++-------------- .../test/ainize-lifecycle.test.js | 8 ++++++++ 4 files changed, 28 insertions(+), 27 deletions(-) diff --git a/scripts/year3-dart100/README.md b/scripts/year3-dart100/README.md index c1521fa..5b3e853 100644 --- a/scripts/year3-dart100/README.md +++ b/scripts/year3-dart100/README.md @@ -4,7 +4,7 @@ Experimental, deployment-specific evidence tooling. It does not claim 100 distin ## Scope -The existing deployment has 100 registered DART task datasets (780 canonical rows) and a public Hugging Face dataset repository, `Minhyun/ainize-dart100-reproduction-20260911`, at revision `9a523ed3268688e90ee18f1ecd93f4fb72a8f056`. The harness verifies the immutable public manifest against all registered IDs and canonical hashes before starting. +The existing deployment has 100 registered DART task datasets (780 canonical rows). The harness verifies the registration evidence, unique dataset IDs and canonical hashes before starting. Publishing a Hugging Face dataset repository is not a prerequisite: the requirement is tool integration, not new Hub publication. The optional historical HF artifact is not a success gate or required network dependency. An HF URL import can supply an Ainize dataset ID through the ordinary teaching API. It adopts a unique matching scratch/balanced lesson, persists submission intent before POST, and polls the same job ID. Ambiguous or missing jobs after an uncertain POST stop the observer rather than authorizing a duplicate submission. Checked READY and NEEDS_MORE drafts are evaluated across all canonical primary and supplied alternate prompts. Execution completion and answer accuracy are separate counters. Failed, cancelled, expired or unchecked lessons remain explicit failures, not dropped denominators. @@ -18,11 +18,11 @@ Node 24, no third-party dependencies: node --test scripts/year3-dart100/test/ainize-lifecycle.test.js ``` -Nine regression tests cover dataset binding, uncertain submissions, job identity collisions, stack ownership, interrupted audits, historical evidence protection, tampering, and canonical denominators. They use fake CLI/model responses and do not establish real GPU training success. +Ten regression tests cover dataset binding without HF publication, optional manifest binding, uncertain submissions, job identity collisions, stack ownership, interrupted audits, historical evidence protection, tampering, and canonical denominators. They use fake CLI/model responses and do not establish real GPU training success. ## Existing deployment -This is not a fresh-machine bootstrap. Current paths are `/mnt/newdata/gov/kpi`, Ainize CLI `/opt/ainize/ainize-cli/dist/bin.js`, and API `http://localhost:3410`. Existing operator credentials, registered datasets, immutable HF publication evidence, and the actual PLE model/trainer are prerequisites. Do not copy secret homes, API tokens or `.env` into this repository. +This is not a fresh-machine bootstrap. Current paths are `/mnt/newdata/gov/kpi`, Ainize CLI `/opt/ainize/ainize-cli/dist/bin.js`, and API `http://localhost:3410`. Existing operator credentials, registered dataset evidence/canonical files, and the actual PLE model/trainer are prerequisites. Do not copy secret homes, API tokens or `.env` into this repository. Place these four JavaScript modules in a frozen source directory under `kpi/evidence//source`, preserve their SHA256 manifest, and execute within the configured Ainize Docker container. The deployment wrapper also snapshots Ainize, serving and trainer image IDs and Docker CPU/GPU/memory limits. @@ -39,4 +39,4 @@ Reuse the same RUN_ID and unchanged snapshot when resuming. The shared flock pre ## Observed limits -The initial live run adopts job `6314e86b-9ba2-4bc3-8a21-e3294663fdd7` without retraining. Its isolated 16-call audit reports 5/8 primary and 1/8 alternate answers correct; it is not a quality pass. A second registered dataset is training under a different job ID. This is not evidence of 100 completed lessons. The running snapshot predates the additional completed-audit resume hash check and unique CLI-error filenames in this source; those additions are regression-tested, not retroactively attributed to that run. +The initial live run adopts job `6314e86b-9ba2-4bc3-8a21-e3294663fdd7` without retraining. Its isolated 16-call audit reports 5/8 primary and 1/8 alternate answers correct; it is not a quality pass. At 2026-09-11 07:10 UTC two datasets have complete inference observations (11/16 primary, 2/16 alternate answers correct in aggregate), and the third job is submitted. This is not evidence of 100 completed lessons. The running immutable snapshot predates the completed-audit resume hash check, unique CLI-error filenames and removal of the old HF publication preflight; those changes are regression-tested, not retroactively attributed to that run. The old snapshot already passed that preflight and does not publish anything while it continues training. diff --git a/scripts/year3-dart100/ainize-lifecycle-state.js b/scripts/year3-dart100/ainize-lifecycle-state.js index 3616ed0..b875b69 100644 --- a/scripts/year3-dart100/ainize-lifecycle-state.js +++ b/scripts/year3-dart100/ainize-lifecycle-state.js @@ -7,13 +7,15 @@ const terminalStatuses = new Set(['READY', 'NEEDS_MORE', 'FAILED', 'CANCELLED', function entriesFrom(registrationBytes, manifest) { const registration = JSON.parse(registrationBytes); - assert.equal(manifest.registrationSha256, sha256(registrationBytes), 'registration hash changed'); assert.equal(registration.datasets.length, 100); - assert.equal(manifest.datasets.length, 100); + if (manifest) { + assert.equal(manifest.registrationSha256, sha256(registrationBytes), 'registration hash changed'); + assert.equal(manifest.datasets.length, 100); + assert.equal(new Set(manifest.datasets.map(dataset => dataset.config)).size, 100, 'duplicate HF config'); + } for (const key of ['datasetId', 'lessonId', 'canonicalSha256']) { assert.equal(new Set(registration.datasets.map(dataset => dataset[key])).size, 100, `duplicate ${key}`); } - assert.equal(new Set(manifest.datasets.map(dataset => dataset.config)).size, 100, 'duplicate HF config'); return registration.datasets.map(dataset => { assert.equal(dataset.ok, true); assert.ok(Object.keys(dataset.checks).length > 0 && Object.values(dataset.checks).every(value => value === true)); @@ -21,11 +23,13 @@ function entriesFrom(registrationBytes, manifest) { assert.ok(/^[A-Za-z0-9_-]+$/.test(dataset.datasetId)); assert.ok(/^[a-f0-9]{64}$/.test(dataset.canonicalSha256)); assert.ok(Number.isInteger(dataset.rows) && dataset.rows > 0); - const published = manifest.datasets.find(entry => entry.config === dataset.lessonId); - assert.ok(published, 'missing HF config'); - assert.equal(published.ainizeDatasetId, dataset.datasetId); - assert.equal(published.sha256, dataset.canonicalSha256); - assert.equal(published.rows, dataset.rows); + if (manifest) { + const configured = manifest.datasets.find(entry => entry.config === dataset.lessonId); + assert.ok(configured, 'missing HF config'); + assert.equal(configured.ainizeDatasetId, dataset.datasetId); + assert.equal(configured.sha256, dataset.canonicalSha256); + assert.equal(configured.rows, dataset.rows); + } return { lessonId: dataset.lessonId, datasetId: dataset.datasetId, sha256: dataset.canonicalSha256, rows: dataset.rows }; }); } diff --git a/scripts/year3-dart100/ainize-lifecycle.js b/scripts/year3-dart100/ainize-lifecycle.js index 531bf1e..e01c672 100644 --- a/scripts/year3-dart100/ainize-lifecycle.js +++ b/scripts/year3-dart100/ainize-lifecycle.js @@ -10,30 +10,19 @@ async function main() { assert.ok(runId && /^[A-Za-z0-9_-]{1,40}$/.test(runId), 'RUN_ID must be 1-40 safe characters'); const output = path.join(root, 'evidence', runId); const registrationRun = process.env.AINIZE_REGISTRATION_RUN || 'ainize_datasets100_20260911'; - const publicationRun = process.env.HF_PUBLICATION_RUN || 'hf_datasets_publish_r2_20260911'; - for (const value of [registrationRun, publicationRun]) assert.ok(/^[A-Za-z0-9_-]+$/.test(value)); + assert.ok(/^[A-Za-z0-9_-]+$/.test(registrationRun)); const registrationBytes = fs.readFileSync(path.join(root, 'evidence', registrationRun, 'progress.json')); - const publication = JSON.parse(fs.readFileSync(path.join(root, 'evidence', publicationRun, 'publication.json'))); - assert.equal(publication.pass, true); - assert.equal(publication.verifiedFiles, 202); - assert.ok(/^[a-f0-9]{40}$/.test(publication.commit)); - assert.ok(/^[A-Za-z0-9_-]+\/[A-Za-z0-9_.-]+$/.test(publication.repoId)); - const manifestBytes = fs.readFileSync(path.join(root, 'evidence', publicationRun, 'download', 'manifest.json')); - const manifest = JSON.parse(manifestBytes); - const publishedResponse = await fetch(`https://huggingface.co/datasets/${publication.repoId}/resolve/${publication.commit}/manifest.json`, { signal: AbortSignal.timeout(60000) }); - assert.ok(publishedResponse.ok, 'public HF manifest unavailable'); - assert.equal(sha256(Buffer.from(await publishedResponse.arrayBuffer())), sha256(manifestBytes), 'public manifest mismatch'); - const entries = entriesFrom(registrationBytes, manifest); + const entries = entriesFrom(registrationBytes); for (const entry of entries) { assert.equal(sha256(fs.readFileSync(path.join(root, 'evidence', registrationRun, `${entry.lessonId}-canonical.jsonl`))), entry.sha256); } - const identity = { runId, registrationRun, registrationSha256: sha256(registrationBytes), repoId: publication.repoId, revision: publication.commit, manifestSha256: sha256(manifestBytes) }; + const identity = { runId, registrationRun, registrationSha256: sha256(registrationBytes) }; const filename = path.join(output, 'progress.json'); let state = { version: 1, identity, startedAt: new Date().toISOString(), attempts: 0, complete: false, entries: entries.map((entry, index) => ({ ...entry, name: `${runId}-${String(index + 1).padStart(3, '0')}`, status: 'PENDING' })) }; if (fs.existsSync(filename)) { state = JSON.parse(fs.readFileSync(filename)); assert.equal(state.version, 1); - assert.deepEqual(state.identity, identity, 'resume inputs changed'); + for (const [key, value] of Object.entries(identity)) assert.equal(state.identity[key], value, 'resume registration inputs changed'); assert.deepEqual(state.entries.map(({ lessonId, datasetId, sha256: hash, rows }) => ({ lessonId, datasetId, sha256: hash, rows })), entries); } state.attempts++; diff --git a/scripts/year3-dart100/test/ainize-lifecycle.test.js b/scripts/year3-dart100/test/ainize-lifecycle.test.js index 5026c2f..3db83f1 100644 --- a/scripts/year3-dart100/test/ainize-lifecycle.test.js +++ b/scripts/year3-dart100/test/ainize-lifecycle.test.js @@ -16,6 +16,14 @@ function registrationFixture() { const entry = { datasetId: 'dataset-1', name: 'run-001', sha256: sha256('rows'), rows: 2 }; const job = { id: 'job-1', name: entry.name, dataset: { id: entry.datasetId, sha256: entry.sha256, rows: 2 }, mode: 'scratch', context_patch_ids: [], training: { effort: 'balanced' }, status: 'TRAINING' }; +test('100 registered datasets require no Hugging Face publication or remote manifest', () => { + const fixture = registrationFixture(); + assert.equal(entriesFrom(fixture.bytes).length, 100); + const registration = JSON.parse(fixture.bytes); + registration.datasets[1].datasetId = registration.datasets[0].datasetId; + assert.throws(() => entriesFrom(Buffer.from(JSON.stringify(registration))), /duplicate datasetId/); +}); + test('exactly 100 unique registered configurations bind to the published hashes', () => { const fixture = registrationFixture(); assert.equal(entriesFrom(fixture.bytes, fixture.manifest).length, 100); From 59d663489773132cb14d34b31dc219449c23a2e7 Mon Sep 17 00:00:00 2001 From: Haechan Date: Fri, 11 Sep 2026 07:53:43 +0000 Subject: [PATCH 3/5] Add native HF100 integration evidence and guarded marketplace deployment --- scripts/year3-dart100/README.md | 31 ++++++ scripts/year3-dart100/ainize-public-proxy.js | 75 ++++++++++++++ .../year3-dart100/docker/hf-cli.Dockerfile | 8 ++ .../docker/run-ainize-lifecycle.sh | 31 ++++++ .../docker/run-ainize-public-proxy.sh | 18 ++++ scripts/year3-dart100/docker/run-hf-cli.sh | 31 ++++++ .../year3-dart100/docker/run-hf-import100.sh | 31 ++++++ .../docker/switch-ainize-market-ledger.sh | 81 +++++++++++++++ ...352\263\204_HF\354\227\260\353\217\231.md" | 98 +++++++++++++++++++ scripts/year3-dart100/import-hf-dart100.js | 65 ++++++++++++ scripts/year3-dart100/record-hf-imports.js | 48 +++++++++ .../year3-dart100/test/hf-import100.test.js | 22 +++++ .../year3-dart100/test/public-catalog.test.js | 71 ++++++++++++++ .../year3-dart100/test/public-proxy.test.js | 62 ++++++++++++ .../year3-dart100/verify-public-catalog.js | 63 ++++++++++++ 15 files changed, 735 insertions(+) create mode 100644 scripts/year3-dart100/ainize-public-proxy.js create mode 100644 scripts/year3-dart100/docker/hf-cli.Dockerfile create mode 100644 scripts/year3-dart100/docker/run-ainize-lifecycle.sh create mode 100644 scripts/year3-dart100/docker/run-ainize-public-proxy.sh create mode 100644 scripts/year3-dart100/docker/run-hf-cli.sh create mode 100644 scripts/year3-dart100/docker/run-hf-import100.sh create mode 100644 scripts/year3-dart100/docker/switch-ainize-market-ledger.sh create mode 100644 "scripts/year3-dart100/docs/Ainize_0-3\353\213\250\352\263\204_HF\354\227\260\353\217\231.md" create mode 100644 scripts/year3-dart100/import-hf-dart100.js create mode 100644 scripts/year3-dart100/record-hf-imports.js create mode 100644 scripts/year3-dart100/test/hf-import100.test.js create mode 100644 scripts/year3-dart100/test/public-catalog.test.js create mode 100644 scripts/year3-dart100/test/public-proxy.test.js create mode 100644 scripts/year3-dart100/verify-public-catalog.js diff --git a/scripts/year3-dart100/README.md b/scripts/year3-dart100/README.md index 5b3e853..eae642f 100644 --- a/scripts/year3-dart100/README.md +++ b/scripts/year3-dart100/README.md @@ -16,10 +16,39 @@ Node 24, no third-party dependencies: ```sh node --test scripts/year3-dart100/test/ainize-lifecycle.test.js +node --test scripts/year3-dart100/test/hf-import100.test.js ``` Ten regression tests cover dataset binding without HF publication, optional manifest binding, uncertain submissions, job identity collisions, stack ownership, interrupted audits, historical evidence protection, tampering, and canonical denominators. They use fake CLI/model responses and do not establish real GPU training success. +Two additional tests cover native HF import binding. `import-hf-dart100.js` calls the real CLI for 100 existing files and checks the immutable source revision, input/upload/canonical hashes, existing dataset ID, row count, `created=false` and absence of a new training job. It is a read/import observation, not new Hub publication or training. The optional historical DART HF repository is an input fixture, not a prerequisite for the separate lifecycle observer. + +The import observer requires an Ainize CLI build with `ainize dataset ` (source release: https://github.com/ainblockchain/ainize-cli/releases/tag/year3-hf-dataset-import-20260911), private operator/teaching home, registration evidence mounted read-only at `/registration`, an empty writable `/evidence`, and this CommonJS source directory. Run it in a resource-limited container with the existing Ainize API reachable at `http://localhost:3410`: + +```sh +node /source/import-hf-dart100.js Minhyun/ainize-dart100-reproduction-20260911 9a523ed3268688e90ee18f1ecd93f4fb72a8f056 +``` + +The deployment wrapper freezes these two source modules, captures Docker image/limits/state, and runs with 2 CPUs, cpuset 0–7, 2 GiB memory, no additional swap and a read-only root filesystem. Completed imports are recorded individually; a failure stops the observer without retraining, cancelling jobs or overwriting a previous run. The live import finished at 2026-09-11 07:30:47 UTC: 100/100 existing dataset IDs, 780 rows, exit zero. All 100 raw response hashes were independently rechecked. This is not 100 completed training jobs. + +`record-hf-imports.js` binds these imports to the separate ten-node AIN experiment chain through the deployment's existing `common.js`/ain-js helpers. It validates registration, raw responses and canonical bytes before recording a manifest hash. Actual transaction: `0xd53cdbd69b2e256234fff2d776a4b3be1613ac7c6c5467d60a168ee1870c5aa9`, block12297; FINALIZED receipt, exact independent node5 readback and node9 block inclusion checked. Manifest SHA256: `72f5e9a43497f04cab876269f6392a01947b4b44a4ef2fd52961c497832004b1`. This records integration evidence, not a new HF publication, training success, sale or incentive settlement. + +## Deployment helpers and public access + +Install the JavaScript helpers in the existing `/mnt/newdata/gov/kpi/harness` and the `docker/` templates in `/mnt/newdata/gov/kpi/docker`; do not run these deployment-specific shell templates from this repository's source folder. They depend on the existing Ainize container/home, private credentials, evidence/results directories, Docker Compose configuration and locally built images. The chain recorder additionally requires the certification deployment's `common.js` and its configured ain-js dependency. This folder is not a clean-machine installer and does not publish Docker images or npm packages. + +- `verify-public-catalog.js` separates actual ID presence from independent LISTED verification. Its default exit gate requires LISTED; `CATALOG_REQUIRE_LISTED=0` checks presence only while still reporting `verificationComplete=false` for ANNOUNCED. +- `ainize-public-proxy.js` exposes only metadata, native P2P and download/payment routes on loopback port3412, upstream3410. It blocks teaching, operator authentication/management and model mutation; removes operator cookies/Bearer and spoofed forwarding headers; and bounds request size/time/connections. It preserves the node's signed entitlement gates, not bypasses them. This is route isolation, not a complete audit of the underlying P2P protocol. The proxy has no secret-home or Docker-socket mount and runs with1CPU/256MiB/read-only/no-extra-swap. Four isolated tests and real-loopback checks cover forwarding, rejected teaching/admin requests, and denied private-draft downloads. +- `switch-ainize-market-ledger.sh` is the guarded, one-off maintenance used after three complete audits. It requires a deliberately stopped matching observer at a no-pending-submission checkpoint, all server jobs terminal and an empty runtime queue/stack. It backs up the secret home outside Git, changes only `ledger.kind`, preserves all job JSON and identity, then requires resuming the same RUN_ID. It refuses active jobs and does not kill/restart GPU containers or the ten-node chain. Do not reuse the historical PID or create a fake pause marker. + +The experiment's Ainize publisher now uses the public marketplace's `local` ledger. The independent AIN performance chain remains running, and evidence anchors use ain-js separately. Local DAG/CREDIT records are not AIN transfers or blockchain incentive settlement. Public HTTPS callback/Funnel enablement and the publisher's rights/permanence consent are still awaiting operator input. No knowledge has been published to the public marketplace by these helpers, and no verification policy has been weakened to fill the catalog. + +All observer/proxy tests: + +```sh +node --test scripts/year3-dart100/test/*.test.js +``` + ## Existing deployment This is not a fresh-machine bootstrap. Current paths are `/mnt/newdata/gov/kpi`, Ainize CLI `/opt/ainize/ainize-cli/dist/bin.js`, and API `http://localhost:3410`. Existing operator credentials, registered dataset evidence/canonical files, and the actual PLE model/trainer are prerequisites. Do not copy secret homes, API tokens or `.env` into this repository. @@ -40,3 +69,5 @@ Reuse the same RUN_ID and unchanged snapshot when resuming. The shared flock pre ## Observed limits The initial live run adopts job `6314e86b-9ba2-4bc3-8a21-e3294663fdd7` without retraining. Its isolated 16-call audit reports 5/8 primary and 1/8 alternate answers correct; it is not a quality pass. At 2026-09-11 07:10 UTC two datasets have complete inference observations (11/16 primary, 2/16 alternate answers correct in aggregate), and the third job is submitted. This is not evidence of 100 completed lessons. The running immutable snapshot predates the completed-audit resume hash check, unique CLI-error filenames and removal of the old HF publication preflight; those changes are regression-tested, not retroactively attributed to that run. The old snapshot already passed that preflight and does not publish anything while it continues training. + +At 07:39 UTC three complete audits report19/24 primary and4/24 alternate answers correct in aggregate, zero all-answer passes. All three jobs, dataset IDs and operator identity survive the guarded Ainize ledger transition unchanged. The same RUN_ID/source resumed as attempt2; job4 `88b3422b-6bb1-4d97-986d-337e9f9331f7` is training. An observer process ended as part of intentional idle maintenance, not because a polling timeout was mistaken for stopped GPU work. diff --git a/scripts/year3-dart100/ainize-public-proxy.js b/scripts/year3-dart100/ainize-public-proxy.js new file mode 100644 index 0000000..a1038e1 --- /dev/null +++ b/scripts/year3-dart100/ainize-public-proxy.js @@ -0,0 +1,75 @@ +const http = require('node:http'); + +const readPaths = new Set(['/healthz', '/readyz', '/api/info', '/api/catalog', '/api/nodes', '/p2p/info', '/p2p/peers', '/p2p/records', '/p2p/blobs', '/p2p/datasets']); +const readPatterns = [ + /^\/api\/patches\/[A-Za-z0-9_-]+(?:\/(?:quote|records|conflicts|dataset(?:\/(?:rows|manifest))?))?$/, + /^\/api\/teacher\/0x[0-9a-fA-F]{40}$/, + /^\/p2p\/blob\/[0-9a-f]{64}$/, + /^\/p2p\/dataset\/[0-9a-f]{64}(?:\/(?:manifest|benchmark))?$/, + /^\/p2p\/payouts\/[A-Za-z0-9_-]+$/, + /^\/x402\/patch\/[A-Za-z0-9_-]+$/, +]; +const writePaths = new Set(['/p2p/hello', '/p2p/records']); + +function publicRequestAllowed(method, target) { + if (!target || target.length > 8192 || !target.startsWith('/') || target.startsWith('//') || /[\\\r\n]/.test(target)) return false; + const pathname = target.split('?')[0]; + if (pathname.includes('%') || pathname.split('/').some(part => part === '.' || part === '..')) return false; + if (method === 'POST') return writePaths.has(pathname); + if (!['GET', 'HEAD', 'OPTIONS'].includes(method)) return false; + return readPaths.has(pathname) || readPatterns.some(pattern => pattern.test(pathname)) || (method === 'OPTIONS' && writePaths.has(pathname)); +} + +function publicHeaders(headers) { + const output = { ...headers }; + const connectionTokens = String(headers.connection || '').split(',').map(value => value.trim().toLowerCase()); + for (const key of [...connectionTokens, 'host', 'connection', 'transfer-encoding', 'upgrade', 'proxy-authorization', 'proxy-authenticate', 'keep-alive', 'trailer', 'te', 'authorization', 'cookie', 'forwarded', 'x-forwarded-for', 'x-forwarded-host', 'x-forwarded-proto', 'x-real-ip']) delete output[key]; + output.connection = 'close'; + return output; +} + +function createPublicProxy({ upstreamPort = 3410, bodyLimit = 2 * 1024 * 1024 } = {}) { + const server = http.createServer({ requestTimeout: 30000, headersTimeout: 10000 }, (request, response) => { + const fail = (status, message) => { + response.writeHead(status, { 'content-type': 'application/json', connection: 'close', 'cache-control': 'no-store' }); + response.end(JSON.stringify({ error: message })); + }; + if (!publicRequestAllowed(request.method, request.url)) return fail(403, 'This endpoint is not exposed by the public marketplace proxy. Teaching, management and model mutation stay local.'); + if (Number(request.headers['content-length']) > bodyLimit) return fail(413, 'request body too large'); + if (request.method === 'OPTIONS') { + response.writeHead(204, { 'access-control-allow-origin': '*', 'access-control-allow-methods': 'GET, HEAD, POST, OPTIONS', 'access-control-allow-headers': 'content-type, x-ainize-auth, x-payment, x-ainize-dataset-intent' }); + return response.end(); + } + const upstream = http.request({ hostname: '127.0.0.1', port: upstreamPort, path: request.url, method: request.method, headers: publicHeaders(request.headers), timeout: 30000 }, incoming => { + const headers = { ...incoming.headers, 'access-control-allow-origin': '*' }; + delete headers['set-cookie']; + response.writeHead(incoming.statusCode || 502, headers); + incoming.on('error', () => response.destroy()); + incoming.pipe(response); + }); + let received = 0; + request.on('data', chunk => { + received += chunk.length; + if (received > bodyLimit) { + request.unpipe(upstream); + if (!response.headersSent) fail(413, 'request body too large'); + upstream.destroy(); + } + }); + request.on('aborted', () => upstream.destroy()); + upstream.on('timeout', () => upstream.destroy(new Error('upstream timeout'))); + upstream.on('error', () => { if (!response.headersSent) fail(502, 'marketplace node unavailable'); }); + response.on('close', () => upstream.destroy()); + request.pipe(upstream); + }); + server.maxConnections = 64; + server.maxHeadersCount = 64; + return server; +} + +if (require.main === module) { + const server = createPublicProxy(); + server.listen(3412, '127.0.0.1', () => process.stdout.write('public marketplace proxy listening on 127.0.0.1:3412; upstream 127.0.0.1:3410\n')); + process.on('SIGTERM', () => server.close(() => process.exit(0))); +} +module.exports = { publicRequestAllowed, publicHeaders, createPublicProxy }; diff --git a/scripts/year3-dart100/docker/hf-cli.Dockerfile b/scripts/year3-dart100/docker/hf-cli.Dockerfile new file mode 100644 index 0000000..308a5f3 --- /dev/null +++ b/scripts/year3-dart100/docker/hf-cli.Dockerfile @@ -0,0 +1,8 @@ +FROM ain-cert-ainize:repro-20260911-r2 +RUN rm -rf /opt/ainize/ainize-cli/src /opt/ainize/ainize-cli/test /opt/ainize/ainize-cli/dist +COPY src /opt/ainize/ainize-cli/src +COPY test /opt/ainize/ainize-cli/test +COPY package.json /opt/ainize/ainize-cli/package.json +RUN cd /opt/ainize/ainize-cli && npm run build +WORKDIR /opt/ainize/ainize-cli +ENTRYPOINT ["node", "/opt/ainize/ainize-cli/dist/bin.js"] diff --git a/scripts/year3-dart100/docker/run-ainize-lifecycle.sh b/scripts/year3-dart100/docker/run-ainize-lifecycle.sh new file mode 100644 index 0000000..e412896 --- /dev/null +++ b/scripts/year3-dart100/docker/run-ainize-lifecycle.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +set -euo pipefail +KPI=$(cd "$(dirname "$0")/.." && pwd) +RUN_ID=${RUN_ID:?set RUN_ID; reuse the same ID to resume} +[[ "$RUN_ID" =~ ^[A-Za-z0-9_-]{1,40}$ ]] || exit 1 +OUT="$KPI/evidence/$RUN_ID" +if [ ! -d "$OUT" ]; then + mkdir "$OUT" "$OUT/source" + for source in ainize-lifecycle.js ainize-lifecycle-state.js ainize-inference-audit.js m5-judge.js; do + cp "$KPI/harness/$source" "$OUT/source/$source" + done + (cd "$OUT/source" && sha256sum *.js) > "$OUT/source.sha256" +fi +(cd "$OUT/source" && sha256sum --check ../source.sha256) +ATTEMPT=$(mktemp -d "$OUT/launch-XXXXXXXX") +docker inspect ain-cert-ainize-node-1 --format '{{json .HostConfig}}' > "$ATTEMPT/limits.json" +docker inspect ain-cert-ainize-node-1 --format '{{json .Image}}' > "$ATTEMPT/image.json" +for container in flashnext flashtrain; do + docker inspect "$container" --format '{{json .HostConfig}}' > "$ATTEMPT/$container-limits.json" + docker inspect "$container" --format '{{json .Image}}' > "$ATTEMPT/$container-image.json" +done +set +e +docker exec -e RUN_ID="$RUN_ID" \ + -e AINIZE_REGISTRATION_RUN="${AINIZE_REGISTRATION_RUN:-ainize_datasets100_20260911}" \ + ain-cert-ainize-node-1 flock --nonblock --no-fork "$KPI/evidence/.ainize-lifecycle.lock" \ + node "$OUT/source/ainize-lifecycle.js" > "$ATTEMPT/stdout.log" 2> "$ATTEMPT/stderr.log" +code=$? +set -e +printf '%s\n' "$code" > "$ATTEMPT/exit-code.txt" +echo "lifecycle observer exit=$code evidence=$OUT; an observer failure does not cancel or resubmit a teach job" +exit "$code" diff --git a/scripts/year3-dart100/docker/run-ainize-public-proxy.sh b/scripts/year3-dart100/docker/run-ainize-public-proxy.sh new file mode 100644 index 0000000..de1e0b9 --- /dev/null +++ b/scripts/year3-dart100/docker/run-ainize-public-proxy.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +set -euo pipefail +KPI=$(cd "$(dirname "$0")/.." && pwd) +RUN_ID=${RUN_ID:?set a new proxy deployment RUN_ID} +[[ "$RUN_ID" =~ ^[A-Za-z0-9_-]+$ ]] || exit 1 +OUT="$KPI/evidence/$RUN_ID" +mkdir "$OUT" "$OUT/source" +cp "$KPI/harness/ainize-public-proxy.js" "$OUT/source/" +(cd "$OUT/source" && sha256sum *.js) > "$OUT/source.sha256" +docker create --name ain-cert-public-proxy --network host --cpus 1 --cpuset-cpus 0-7 \ + --memory 256m --memory-swap 256m --read-only --cap-drop ALL --security-opt no-new-privileges \ + --user "$(id -u):$(id -g)" --mount "type=bind,src=$OUT/source,dst=/source,readonly" \ + --label org.ain.cert.role=public-marketplace-proxy --entrypoint node \ + ain-cert-ainize-cli:hf-import-20260911-r5 /source/ainize-public-proxy.js > "$OUT/container-id.txt" +docker inspect ain-cert-public-proxy --format '{{json .HostConfig}}' > "$OUT/limits.json" +docker inspect ain-cert-public-proxy --format '{{json .Image}}' > "$OUT/image.json" +docker start ain-cert-public-proxy +docker inspect ain-cert-public-proxy --format '{{json .State}}' > "$OUT/state.json" diff --git a/scripts/year3-dart100/docker/run-hf-cli.sh b/scripts/year3-dart100/docker/run-hf-cli.sh new file mode 100644 index 0000000..9028a25 --- /dev/null +++ b/scripts/year3-dart100/docker/run-hf-cli.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +set -euo pipefail +KPI=$(cd "$(dirname "$0")/.." && pwd) +RUN_ID=${RUN_ID:-hf_cli_$(date -u +%Y%m%dT%H%M%SZ)} +[[ "$RUN_ID" =~ ^[A-Za-z0-9_-]+$ ]] || exit 1 +OUT="$KPI/evidence/$RUN_ID" +HOME_DIR=${AINIZE_HOME:-$KPI/ainize/home-docker} +IMAGE=${HF_CLI_IMAGE:-ain-cert-ainize-cli:hf-import-20260911-r5} +mkdir "$OUT" +container="ain-cert-$RUN_ID" +docker create --name "$container" --network host --cpus 2 --cpuset-cpus 0-7 \ + --memory 2g --memory-swap 2g --read-only --tmpfs /tmp:rw,nosuid,size=256m \ + --user "$(id -u):$(id -g)" -e AINIZE_HOME="$HOME_DIR" \ + --mount "type=bind,src=$HOME_DIR,dst=$HOME_DIR" \ + "$IMAGE" "$@" > "$OUT/container-id.txt" +docker inspect "$container" --format '{{json .HostConfig}}' > "$OUT/limits.json" +docker inspect "$container" --format '{{json .Image}}' > "$OUT/image.json" +set +e +docker start -a "$container" > "$OUT/stdout.log" 2> "$OUT/stderr.log" +set -e +docker inspect "$container" --format '{{json .State}}' > "$OUT/state.json" +running=$(docker inspect "$container" --format '{{.State.Running}}') +if [ "$running" = true ]; then + echo "CLI container is still running: $container; inspect the same container before retrying" >&2 + exit 1 +fi +code=$(docker inspect "$container" --format '{{.State.ExitCode}}') +docker rm "$container" >/dev/null +cat "$OUT/stdout.log" +cat "$OUT/stderr.log" >&2 +exit "$code" diff --git a/scripts/year3-dart100/docker/run-hf-import100.sh b/scripts/year3-dart100/docker/run-hf-import100.sh new file mode 100644 index 0000000..15c553f --- /dev/null +++ b/scripts/year3-dart100/docker/run-hf-import100.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +set -euo pipefail +KPI=$(cd "$(dirname "$0")/.." && pwd) +RUN_ID=${RUN_ID:?set a new RUN_ID} +[[ "$RUN_ID" =~ ^[A-Za-z0-9_-]+$ ]] || exit 1 +OUT="$KPI/evidence/$RUN_ID" +HOME_DIR=${AINIZE_HOME:-$KPI/ainize/home-docker} +REGISTRATION_RUN=${AINIZE_REGISTRATION_RUN:-ainize_datasets100_20260911} +[[ "$REGISTRATION_RUN" =~ ^[A-Za-z0-9_-]+$ ]] || exit 1 +IMAGE=${HF_CLI_IMAGE:-ain-cert-ainize-cli:hf-import-20260911-r5} +mkdir "$OUT" "$OUT/source" +cp "$KPI/harness/import-hf-dart100.js" "$KPI/harness/ainize-lifecycle-state.js" "$OUT/source/" +(cd "$OUT/source" && sha256sum *.js) > "$OUT/source.sha256" +container="ain-cert-$RUN_ID" +docker create --name "$container" --network host --cpus 2 --cpuset-cpus 0-7 \ + --memory 2g --memory-swap 2g --read-only --tmpfs /tmp:rw,nosuid,size=256m \ + --user "$(id -u):$(id -g)" -e AINIZE_HOME="$HOME_DIR" \ + --mount "type=bind,src=$HOME_DIR,dst=$HOME_DIR" \ + --mount "type=bind,src=$KPI/evidence/$REGISTRATION_RUN,dst=/registration,readonly" \ + --mount "type=bind,src=$OUT,dst=/evidence" \ + --mount "type=bind,src=$OUT/source,dst=/source,readonly" \ + --entrypoint node "$IMAGE" /source/import-hf-dart100.js "$@" > "$OUT/container-id.txt" +docker inspect "$container" --format '{{json .HostConfig}}' > "$OUT/limits.json" +docker inspect "$container" --format '{{json .Image}}' > "$OUT/image.json" +docker start -a "$container" > "$OUT/stdout.log" 2> "$OUT/stderr.log" || true +docker inspect "$container" --format '{{json .State}}' > "$OUT/state.json" +if [ "$(docker inspect "$container" --format '{{.State.Running}}')" = true ]; then + echo "Importer still running: $container; observe it instead of starting another" >&2 + exit 1 +fi +exit "$(docker inspect "$container" --format '{{.State.ExitCode}}')" diff --git a/scripts/year3-dart100/docker/switch-ainize-market-ledger.sh b/scripts/year3-dart100/docker/switch-ainize-market-ledger.sh new file mode 100644 index 0000000..e14fa4c --- /dev/null +++ b/scripts/year3-dart100/docker/switch-ainize-market-ledger.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +set -euo pipefail +KPI=$(cd "$(dirname "$0")/.." && pwd) +RUN_ID=${RUN_ID:?use the evidence directory containing observer-pause.json} +[[ "$RUN_ID" =~ ^[A-Za-z0-9_-]+$ ]] || exit 1 +OUT="$KPI/evidence/$RUN_ID" +BACKUP="$KPI/secrets/$RUN_ID" +test -f "$OUT/observer-pause.json" +test ! -e "$BACKUP" +bash "$KPI/docker/ainize-cli.sh" teach jobs --json > "$OUT/jobs-before.json" +curl --fail --silent --show-error http://localhost:3410/api/info > "$OUT/info-before.json" +node - "$KPI" "$OUT" <<'JS' +const fs = require('fs'); +const assert = require('assert/strict'); +const [root, output] = process.argv.slice(2); +const pause = JSON.parse(fs.readFileSync(`${output}/observer-pause.json`)); +const proc = fs.readFileSync(`/proc/${pause.pid}/status`, 'utf8'); +assert.match(proc, /^State:\s+T/m, 'observer must be deliberately stopped, not an active submitter'); +assert.equal(fs.readFileSync(`/proc/${pause.pid}/cmdline`, 'utf8'), `node\0${root}/evidence/ainize_lifecycle100_20260911/source/ainize-lifecycle.js\0`); +const progress = JSON.parse(fs.readFileSync(`${root}/evidence/ainize_lifecycle100_20260911/progress.json`)); +assert.equal(progress.summary.jobs, progress.summary.inferenceComplete, 'unfinished observed job'); +assert.ok(progress.entries.every(entry => entry.done || (!entry.jobId && !entry.submissionIntent)), 'ambiguous pending submission'); +const jobs = JSON.parse(fs.readFileSync(`${output}/jobs-before.json`)); +const terminal = new Set(['READY', 'NEEDS_MORE', 'FAILED', 'CANCELLED', 'REJECTED', 'ANNOUNCED', 'PENDING_REVIEW', 'EXPIRED']); +assert.ok(Array.isArray(jobs.items) && jobs.items.length < 500); +assert.ok(jobs.items.every(job => terminal.has(job.status)), 'a server-side job is active; do not stop the node'); +const info = JSON.parse(fs.readFileSync(`${output}/info-before.json`)); +assert.equal(info.ledger.kind, 'ain'); +assert.equal(info.runtime.available, true); +const queue = info.runtime.queue; +assert.ok(queue && !queue.running && !queue.waiting && !queue.lock && !queue.queued?.length); +assert.deepEqual(info.runtime.applied, []); +const config = JSON.parse(fs.readFileSync(`${root}/ainize/home-docker/config.json`)); +assert.equal(config.ledger.kind, 'ain'); +assert.ok(config.peers.includes('https://ainize.ai')); +fs.writeFileSync(`${output}/config-before.sha256`, require('crypto').createHash('sha256').update(fs.readFileSync(`${root}/ainize/home-docker/config.json`)).digest('hex') + '\n', {flag:'wx'}); +JS +mkdir -m 700 "$BACKUP" +docker inspect ain-cert-ainize-node-1 --format '{{json .Image}}' > "$OUT/image-before.json" +export AINIZE_UID="$(id -u)" AINIZE_GID="$(id -g)" DOCKER_GID="$(stat -c %g /var/run/docker.sock)" +docker compose -f "$KPI/docker/compose.ainize.json" stop -t 60 node > "$OUT/stop.log" 2>&1 +umask 077 +tar -C "$KPI/ainize" -cf "$BACKUP/home-docker.tar" home-docker +cp "$KPI/ainize/home-docker/config.json" "$BACKUP/config-before.json" +umask 022 +sha256sum "$BACKUP/home-docker.tar" > "$OUT/backup.sha256" +RUN_ID="${RUN_ID}_config" bash "$KPI/docker/run-hf-cli.sh" config set ledger.kind local --json > "$OUT/config-set.json" +node - "$KPI" "$BACKUP" <<'JS' +const fs = require('fs'); +const assert = require('assert/strict'); +const [root, backup] = process.argv.slice(2); +const before = JSON.parse(fs.readFileSync(`${backup}/config-before.json`)); +const after = JSON.parse(fs.readFileSync(`${root}/ainize/home-docker/config.json`)); +assert.equal(after.ledger.kind, 'local'); +after.ledger.kind = before.ledger.kind; +assert.deepEqual(after, before, 'only ledger.kind may change'); +JS +docker compose -f "$KPI/docker/compose.ainize.json" up -d --no-deps node > "$OUT/start.log" 2>&1 +for attempt in {1..60}; do + if curl --fail --silent --max-time 5 http://localhost:3410/readyz > "$OUT/ready.json"; then break; fi + sleep 2 +done +curl --fail --silent --show-error http://localhost:3410/api/info > "$OUT/info-after.json" +bash "$KPI/docker/ainize-cli.sh" teach jobs --json > "$OUT/jobs-after.json" +docker inspect ain-cert-ainize-node-1 --format '{{json .HostConfig}}' > "$OUT/limits-after.json" +node - "$OUT" <<'JS' +const fs = require('fs'); +const assert = require('assert/strict'); +const output = process.argv[2]; +const before = JSON.parse(fs.readFileSync(`${output}/jobs-before.json`)); +const after = JSON.parse(fs.readFileSync(`${output}/jobs-after.json`)); +assert.deepEqual(after, before, 'the same teaching jobs and dataset IDs must survive'); +const oldInfo = JSON.parse(fs.readFileSync(`${output}/info-before.json`)); +const newInfo = JSON.parse(fs.readFileSync(`${output}/info-after.json`)); +assert.equal(newInfo.ledger.kind, 'local'); +assert.equal(newInfo.node.address, oldInfo.node.address); +assert.equal(newInfo.runtime.available, true); +assert.deepEqual(newInfo.runtime.applied, []); +fs.writeFileSync(`${output}/result.json`, JSON.stringify({at:new Date().toISOString(),pass:true,jobsPreserved:after.items.length,node:newInfo.node.address,from:'ain',to:'local',scope:'Ainize publication ledger only; the ten-node AIN performance chain and all GPU containers remain unchanged. Local DAG/CREDIT records are not AIN transactions. Resume the same lifecycle RUN_ID after maintenance.'},null,2)+'\n',{flag:'wx'}); +JS +echo "Ainize now uses the public marketplace's local ledger; jobs preserved. Resume the same lifecycle RUN_ID when maintenance is complete." diff --git "a/scripts/year3-dart100/docs/Ainize_0-3\353\213\250\352\263\204_HF\354\227\260\353\217\231.md" "b/scripts/year3-dart100/docs/Ainize_0-3\353\213\250\352\263\204_HF\354\227\260\353\217\231.md" new file mode 100644 index 0000000..fe56e27 --- /dev/null +++ "b/scripts/year3-dart100/docs/Ainize_0-3\353\213\250\352\263\204_HF\354\227\260\353\217\231.md" @@ -0,0 +1,98 @@ +# Ainize 0–3단계와 Hugging Face 연동 + +## 최신 진행: 2026-09-11 07:47 UTC + +- HF native CLI 가져오기가 **100/100 데이터셋·780행** 완료되었다. 모든 원문 로그 SHA를 재검증했고 기존 Ainize ID/정규 해시가 같으며 새 학습 job은 만들지 않았다. `hf_native_dart100_20260911`, exit0이다. +- 실제 teach/추론은3개 완료, 기본19/24·대체4/24, 전 문항 정답 데이터셋0개다. 네 번째 job `88b3422b-6bb1-4d97-986d-337e9f9331f7`이 학습 중이다. HF100개 가져오기를100개 학습 완료로 보고하지 않는다. +- 원장 불일치를 해결했다. 세 번째 추론 감사가 끝난 체크포인트에서 관측기만 멈추고 서버의 모든 job terminal·모델 queue/stack 비어 있음을 검증했다. 운영 홈을0700/0600 비공개로 백업하고 **Ainize의 ledger.kind만 ain→local**로 전환했다. node identity/운영자 인증/데이터셋/job3개의 전체 JSON이 전후 동일함을 확인했다. 같은 RUN_ID/같은 소스 스냅샷을 새 observer로 재개했고 중복 학습하지 않았다. +- **10개 AIN 성능시험 체인과 GPU 컨테이너는 변경하지 않았다.** Ainize 공개 마켓플레이스는 local DAG/CREDIT를 사용하고, 데이터/실험 증빙은 ain-js로 기존 AIN 체인에 따로 기록한다. local 거래를 AIN 전송/정산이라고 주장하지 않는다. 네이티브 AIN 인센티브 실증은 여전히 별도 미완료 항목이다. +- 현재 공개 HTTPS callback은 아직 없다. `127.0.0.1:3412`에 공개용 제한 프록시를 Docker1CPU/256MiB/read-only로 구동했다. 메타데이터·P2P·다운로드만 전달하며 학습/관리/모델변경 API와 운영자 쿠키·Bearer 전달은 차단한다. 실제 API 검사에서 학습/초기 관리자 설정403, 비공개 초안 무인증 다운로드402를 확인했다. 이것이 전체 P2P 보안 감사를 뜻하지는 않는다. +- Tailscale Funnel은 tailnet에서 비활성화되어 사용자에게 장치 활성화 또는 관리자 HTTPS 프록시를 요청했다. 외부 URL이 실제 도달 가능한지 확인하기 전에 노드 endpoint를 허위 URL로 바꾸지 않는다. 영구공개/배포권 동의도 사용자 확인을 요청했으며 자동 동의하지 않는다. 아직 `teach publish` 완료/공개 목록 노출/구매·추론 완료는 아니다. + +아래 원장 불일치 설명은 최초 발견 당시의 기록이며 현재 불일치 자체는 해결되었다. 유지보수 증빙은 `kpi/evidence/ainize_market_ledger_transition_20260911/`, 프록시 증빙은 `kpi/evidence/ainize_public_proxy_20260911/`에 있다. + +## 범위 정정 + +사용자 확인 기준은 **기존 HF 데이터셋 URL 연동 → Ainize dataset → teach → publish → use/chat**이다. HF dataset 저장소를 새로 게시하는 것은 요구사항이 아니다. 이미 게시한 DART 묶음은 선택적인 입력/과거 증빙으로만 보존하며 새 HF 게시를 성공 조건으로 요구하지 않는다. 데이터셋100개, 학습 job100개, LISTED100개, 기반 모델100종은 다른 계수다. + +## 0. 참여 + +새 참여 노드에서 다음 흐름을 사용한다. `init`은 기본적으로 새 키를 생성한다. 기존 키를 사용해야 할 때만 사용자 제시 `--private-key `를 사용하며 키를 로그·Git에 남기지 않는다. + +```sh +ainize init --name my-node --peer https://ainize.ai +ainize start -d && ainize login +``` + +실제 학습에는 호환되는 모델 서빙 API와 트레이너 설정이 필요하다. peer 연결만으로 GPU나 학습 서비스가 자동 제공되지는 않는다. 원장 종류와 외부에서 접근 가능한 자기 노드 endpoint도 확인한다. + +최초 실험 홈은 이미 초기화된 AIN 노드였으므로 위 명령으로 덮지 않았다. 대신 `bash kpi/docker/ainize-cli.sh peers add https://ainize.ai --json`을 실행해 **공개 local 원장 ↔ 실험 AIN 원장 불일치** 경고를 확인했다. 이후 위 최신 진행처럼 유휴 체크포인트에서 설정1개만 전환했다. `init --force`는 런타임/학습 설정까지 초기값으로 재작성하므로 사용하지 않았다. 진행 중인 학습을 취소하거나 모델을 재시작하지 않는다. + +## 1. 데이터셋 연동과 가르치기 + +로컬 파일 또는 HF URL 중 하나를 입력한다. + +```sh +ainize teach dataset upload questions.csv +ainize dataset https://huggingface.co/datasets/owner/questions --config default --split train +ainize teach train --key-file /private/teaching-key --wait +``` + +`owner/questions`, dataset ID, 키 파일 경로는 실제 값으로 대체한다. `--key `도 지원하지만 실험에서는 비밀을 명령 인수에 넣지 않는 키 파일을 우선한다. 이미 있는 동일 dataset/hash의 job은 다시 제출하지 않고 `teach status `로 관측한다. + +HF URL 명령은 이번 CLI 변경으로 구현했다. 저장소/viewer/resolve URL을 지원하고 revision을 SHA로 고정하며 페이지별 revision·행 연속성·누락/잘림을 검사한다. `--columns`로 JSON/JSONL 컬럼을 Ainize prompt/answer로 변환한다. 기본은 선택 split 전체(최대10,000행), 초과 시 명시적 `--limit`가 필요하고 샘플 범위를 출력한다. Viewer가 준비되지 않았으면 `--file`로 고정리비전의 JSONL/JSON/CSV/TSV/TXT를 읽는다. 32MiB 가져오기 제한과 노드의 행수/PII/형식 제한은 별도이며 우회하지 않는다. Viewer 페이지는 [HF 공식 rows 문서](https://huggingface.co/docs/dataset-viewer/rows)의 최대100행 요청과 [splits/config 문서](https://huggingface.co/docs/dataset-viewer/splits)를 사용한다. 이100행 페이지 크기는 요구사항의 데이터셋100개와 무관하다. + +현재 Docker 실환경에서 실행 가능한 예: + +```bash +cd /mnt/newdata/gov +RUN_ID=hf_external_$(date -u +%Y%m%dT%H%M%SZ) bash kpi/docker/run-hf-cli.sh \ + dataset https://huggingface.co/datasets/lhoestq/demo1 --split train \ + --columns '{"prompt":"review","answer":"star"}' --node http://localhost:3410 --json +RUN_ID=hf_dart_$(date -u +%Y%m%dT%H%M%SZ) bash kpi/docker/run-hf-cli.sh \ + dataset https://huggingface.co/datasets/Minhyun/ainize-dart100-reproduction-20260911 \ + --revision 9a523ed3268688e90ee18f1ecd93f4fb72a8f056 \ + --file data/dart-001-company_ceo_nm.jsonl --node http://localhost:3410 --json +``` + +이 명령은 HF를 읽고 현재 Ainize에 등록만 한다. 기본값으로 학습·HF 게시·마켓플레이스 발행을 하지 않는다. 원문, 컬럼변환 후 업로드, Ainize 정규화 결과의 해시를 구분한다. 비공개 HF 토큰은 필요한 경우에만0600파일로 제공하며 컨테이너 안에서 접근 가능한 비공개 경로여야 한다. 토큰은 Ainize API나 출처 JSON에 전달하지 않는다. + +실측: 외부 `lhoestq/demo1` train 전체5행 → dataset `4400c876-72ff-4004-8eb2-e8ebbba49685`, accepted5/rejected0. 이 보조 시험은 DART100개 계수에 포함하지 않는다. DART 첫 파일은 기존 ID `708fedfb-503d-4124-b12e-b3e85dfe73e3`·SHA `405249ef16b48a3754b481092a968d004996acafd9c60f72fd06b655fe80f8c4`와 같으며 created=false였다. 해당8행 또는 외부5행을 별도 데이터셋8개/5개로 세지 않는다. 최초 컬럼변환 실패 로그도 보존했다. + +100개 고정 파일을 새 학습 없이 기존 ID/해시와 대조하는 명령: + +```bash +RUN_ID=hf_native_dart100_20260911 bash kpi/docker/run-hf-import100.sh \ + Minhyun/ainize-dart100-reproduction-20260911 9a523ed3268688e90ee18f1ecd93f4fb72a8f056 +``` + +위 실제 RUN_ID는100개 관측이 완료되었으므로 덮어쓰거나 재실행하지 않는다. 새 관측에는 새 RUN_ID가 필요하다. 결과는 `kpi/evidence//progress.json`이며 완료 전100개 연동 성공으로 보고하지 않는다. 데이터셋 지원 증빙은 등록100개와 이 연동 관측, 별도 `ainize_lifecycle100_20260911`의 같은 dataset ID에 연결된 실제 teach/추론 결과를 함께 본다. + +## 2. 발행 + +```sh +ainize teach publish --name "지식 이름" --consent-permanent --consent-rights +``` + +현재 CLI는 사용자 축약 예시의 job ID 외에 이름과 두 동의 플래그를 요구한다. 영구 공개/배포 권리는 게시자가 확인해야 하며 에이전트가 자동 동의하거나 PII 게이트를 우회하지 않는다. READY/checks 및 운영자 review 결과를 확인하고, 실제 반환된 published knowledge ID를 기록한다. 공개 카탈로그는 ANNOUNCED도 표시할 수 있다. **목록 노출과 독립 검증 완료(LISTED)를 구별**하여 기록하며, 사용자가 요청한 노출 자체에 별도 quorum 완료 조건을 덧붙이지 않는다. HF 저장소 게시와는 다른 발행이다. + +공개 노출만 관측할 때는 `CATALOG_REQUIRE_LISTED=0 EXPECTED_PATCH_IDS=<실제ID> RUN_ID=<새ID> node kpi/harness/verify-public-catalog.js`를 사용한다. `catalogPresenceComplete`와 `verificationComplete`를 따로 보고한다. 기본 모드는 이전처럼 LISTED까지 검사하며, 이 검증기는 미검증 구매를 승인하거나 판매 정책을 변경하지 않는다. + +## 3. 구매·추론 + +```sh +ainize use +ainize chat "질문" +``` + +견적·결제/무료 여부·다운로드 해시·호환 모델에 실제 로드된 스택과 적용 전후 응답을 보존한다. 판매자 자기 구매나 무료 다운로드를 유상 독립 구매 증빙으로 둔갑시키지 않는다. 구매는 검증 상태·판매자 정책·지갑/잔액에 따르며 임의 지출 한도를 만들지 않는다. 최신 upstream CLI는 미검증 지식 구매에 경고와 별도 확인을 요구한다. 관측 성공을 위해 `--yes`로 무조건 우회하지 않는다. 공개 노출/LISTED만으로 내 모델 로드 또는 정답 성공까지 주장하지 않는다. + +## 공개 관리자에게 전달할 확인 사항 + +2026-09-11 공개 `/api/info`는 local 원장·Qwen3.8-Flash-Next runtime.available=true·quorum2, `/api/catalog`는 total0이다. 실험 HTTP3410 노드도 현재 local 원장으로 맞췄으며 loopback만 수신한다. 첫 peer 추가 증빙은 `kpi/evidence/ainize_public_peer_join_20260911/`에 있다. 다음 정보는 비밀키나 관리자 비밀번호 없이 전달 가능하다. + +1. 원장 종류 불일치는 해결되었으므로 공개 서버의 원장을 재초기화할 필요는 없다. 같은 local DAG의 서명 기록 수신과 네트워크 버전 호환을 확인한다. +2. 공개 서버/구매자/검증자가 이 머신의 **제한 프록시127.0.0.1:3412**에 연결할 HTTPS endpoint 또는 역방향 프록시 경로를 알려 달라. Tailscale Funnel을 이 장치에 허용하는 방법도 가능하다. `localhost:3410`은 원격 서버에서 이 머신을 가리키지 않는다. incoming hello가 peer의 last_seen/failures를 갱신하므로 reachable 표시 하나만으로 원격의 역방향 파일 다운로드가 성공했다고 판단하지 않는다. +3. LISTED 검증 완료까지 증빙하기 위해 같은 원장에서 Qwen3.8 패치를 실제 검증할 독립 verifier 최소2개의 주소·도달성·런타임 호환 상태를 확인해 달라. 이는 단순 목록 노출과 구별한다. peer 개수만으로 유효한 독립 검증 완료를 주장하지 않는다. + +공개 `teach/policy.enabled=false`는 현재 원격 공개 서버에 직접 가르치기는 꺼져 있다는 뜻이다. 자기 참여 노드에서 가르치는 사용자 흐름에는 이를 반드시 켤 필요가 없다. 위 환경 연결과 게시자 동의가 확인되기 전에는 공개 발행·결제·목록 노출을 완료라고 보고하지 않는다. diff --git a/scripts/year3-dart100/import-hf-dart100.js b/scripts/year3-dart100/import-hf-dart100.js new file mode 100644 index 0000000..f825d13 --- /dev/null +++ b/scripts/year3-dart100/import-hf-dart100.js @@ -0,0 +1,65 @@ +const fs = require('fs'); +const path = require('path'); +const assert = require('assert/strict'); +const { spawnSync } = require('child_process'); +const { entriesFrom, sha256 } = require('./ainize-lifecycle-state'); + +function verifyImport(entry, result, repository, revision) { + assert.equal(result.source.repository, repository); + assert.equal(result.source.revision, revision); + assert.equal(result.source.file, `data/${entry.lessonId}.jsonl`); + assert.equal(result.source.inputSha256, entry.sha256); + assert.equal(result.source.sha256, entry.sha256); + assert.equal(result.dataset_id, entry.datasetId); + assert.equal(result.dataset.id, entry.datasetId); + assert.equal(result.dataset.sha256, entry.sha256); + assert.equal(result.dataset.rows, entry.rows); + assert.equal(result.created, false, 'expected reuse of the existing DART dataset'); + assert.equal(result.job, undefined, 'import must not train'); +} + +async function main() { + const [repository, revision] = process.argv.slice(2); + assert.ok(/^[A-Za-z0-9_.-]+\/[A-Za-z0-9_.-]+$/.test(repository || '')); + assert.ok(/^[a-f0-9]{40}$/.test(revision || ''), 'pass an immutable HF commit SHA'); + const registrationBytes = fs.readFileSync('/registration/progress.json'); + const entries = entriesFrom(registrationBytes); + const output = '/evidence'; + const filename = path.join(output, 'progress.json'); + assert.ok(!fs.existsSync(filename), 'use a new RUN_ID; preserve previous observations'); + const state = { repository, revision, registrationSha256: sha256(registrationBytes), startedAt: new Date().toISOString(), target: entries.length, complete: false, entries: [] }; + const save = () => { + state.updatedAt = new Date().toISOString(); + fs.writeFileSync(`${filename}.tmp`, JSON.stringify(state, null, 2) + '\n'); + fs.renameSync(`${filename}.tmp`, filename); + }; + save(); + try { + for (const entry of entries) { + const canonical = fs.readFileSync(`/registration/${entry.lessonId}-canonical.jsonl`); + assert.equal(sha256(canonical), entry.sha256); + const args = ['/opt/ainize/ainize-cli/dist/bin.js', 'dataset', `https://huggingface.co/datasets/${repository}`, '--revision', revision, '--file', `data/${entry.lessonId}.jsonl`, '--node', 'http://localhost:3410', '--json']; + const result = spawnSync(process.execPath, args, { encoding: 'utf8', maxBuffer: 8 * 1024 * 1024, timeout: 180000 }); + fs.writeFileSync(path.join(output, `${entry.lessonId}.json`), result.stdout || '', { flag: 'wx' }); + fs.writeFileSync(path.join(output, `${entry.lessonId}.stderr.log`), result.stderr || '', { flag: 'wx' }); + assert.ifError(result.error); + assert.equal(result.status, 0, `CLI import failed: ${entry.lessonId}`); + const imported = JSON.parse(result.stdout); + verifyImport(entry, imported, repository, revision); + state.entries.push({ ...entry, rawSha256: sha256(result.stdout), reused: true }); + save(); + process.stdout.write(`${state.entries.length}/${state.target} ${entry.lessonId}\n`); + if (state.entries.length < entries.length) await new Promise(resolve => setTimeout(resolve, 6500)); + } + state.complete = true; + state.scope = '100 existing HF files imported through the native Ainize CLI and bound to existing DART dataset IDs; no Hub publication, no training, no marketplace listing'; + save(); + } catch (error) { + state.error = error.message; + save(); + throw error; + } +} + +if (require.main === module) main().catch(error => { process.stderr.write(`${error.stack}\n`); process.exitCode = 1; }); +module.exports = { verifyImport }; diff --git a/scripts/year3-dart100/record-hf-imports.js b/scripts/year3-dart100/record-hf-imports.js new file mode 100644 index 0000000..67ed052 --- /dev/null +++ b/scripts/year3-dart100/record-hf-imports.js @@ -0,0 +1,48 @@ +const fs = require('fs'); +const path = require('path'); +const assert = require('assert/strict'); +const { entriesFrom, sha256 } = require('./ainize-lifecycle-state'); +const { verifyImport } = require('./import-hf-dart100'); +const { KPI_DIR, APP, newAin, envSnapshot, assertResultFree, assertChainPathFree, recordAndVerifyFinal, writeResult } = require('./common'); + +async function main() { + const runId = process.env.RUN_ID; + const [importRun, registrationRun = 'ainize_datasets100_20260911'] = process.argv.slice(2); + for (const value of [runId, importRun, registrationRun]) assert.ok(value && /^[A-Za-z0-9_-]+$/.test(value)); + assertResultFree(`hf-imports-${runId}`); + const root = path.join(KPI_DIR, 'evidence', importRun); + const registrationBytes = fs.readFileSync(path.join(KPI_DIR, 'evidence', registrationRun, 'progress.json')); + const expected = entriesFrom(registrationBytes); + const progressBytes = fs.readFileSync(path.join(root, 'progress.json')); + const progress = JSON.parse(progressBytes); + assert.equal(progress.registrationSha256, sha256(registrationBytes)); + assert.equal(progress.complete, true); + assert.equal(progress.entries.length, 100); + const manifest = expected.map((entry, index) => { + const saved = progress.entries[index]; + for (const key of ['lessonId', 'datasetId', 'sha256', 'rows']) assert.equal(saved[key], entry[key]); + const rawBytes = fs.readFileSync(path.join(root, `${entry.lessonId}.json`)); + assert.equal(sha256(rawBytes), saved.rawSha256); + verifyImport(entry, JSON.parse(rawBytes), progress.repository, progress.revision); + const canonical = fs.readFileSync(path.join(KPI_DIR, 'evidence', registrationRun, `${entry.lessonId}-canonical.jsonl`)); + assert.equal(sha256(canonical), entry.sha256); + return { ...entry, importEvidenceSha256: saved.rawSha256 }; + }); + const state = JSON.parse(fs.readFileSync(path.join(root, 'state.json'))); + assert.equal(state.Running, false); + assert.equal(state.ExitCode, 0); + const manifestBytes = Buffer.from(JSON.stringify(manifest)); + fs.writeFileSync(path.join(KPI_DIR, 'evidence', runId, 'manifest.json'), manifestBytes, { flag: 'wx' }); + const environment = await envSnapshot(); + const ain = newAin(3, null, 13); + const target = `/apps/${APP}/hf_dataset_integrations/${runId}`; + await assertChainPathFree(ain, target); + const value = { repository: progress.repository, revision: progress.revision, datasets: 100, rows: manifest.reduce((total, entry) => total + entry.rows, 0), + importRun, registrationRun, registrationSha256: sha256(registrationBytes), progressSha256: sha256(progressBytes), manifestSha256: sha256(manifestBytes), + scope: 'Existing HF data imported by the native Ainize CLI and bound to 100 existing dataset IDs; no Hub publication, no claim of 100 completed teach/inference jobs or public marketplace sales' }; + const verified = await recordAndVerifyFinal(ain, target, value); + await writeResult(`hf-imports-${runId}`, { value, target, verified, pass: true }, environment); + console.log(JSON.stringify({ ...value, txHash: verified.txHash, block: verified.blockNumber })); +} + +main().catch(error => { console.error(error); process.exitCode = 1; }); diff --git a/scripts/year3-dart100/test/hf-import100.test.js b/scripts/year3-dart100/test/hf-import100.test.js new file mode 100644 index 0000000..98a03bd --- /dev/null +++ b/scripts/year3-dart100/test/hf-import100.test.js @@ -0,0 +1,22 @@ +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const { verifyImport } = require('../import-hf-dart100'); + +function fixture() { + const entry = { lessonId: 'dart-001-company_ceo_nm', datasetId: 'dataset-1', sha256: '1'.repeat(64), rows: 8 }; + const result = { source: { repository: 'owner/data', revision: '2'.repeat(40), file: `data/${entry.lessonId}.jsonl`, inputSha256: entry.sha256, sha256: entry.sha256 }, dataset_id: entry.datasetId, dataset: { id: entry.datasetId, sha256: entry.sha256, rows: entry.rows }, created: false }; + return { entry, result }; +} + +test('native import must reuse the exact existing dataset without training', () => { + const { entry, result } = fixture(); + verifyImport(entry, result, 'owner/data', '2'.repeat(40)); +}); + +test('different sources, hashes, IDs, row counts and accidental training fail', () => { + for (const mutate of [result => { result.source.revision = '3'.repeat(40); }, result => { result.source.inputSha256 = '0'.repeat(64); }, result => { result.dataset_id = 'wrong'; }, result => { result.dataset.rows = 7; }, result => { result.created = true; }, result => { result.job = { id: 'new-job' }; }]) { + const { entry, result } = fixture(); + mutate(result); + assert.throws(() => verifyImport(entry, result, 'owner/data', '2'.repeat(40))); + } +}); diff --git a/scripts/year3-dart100/test/public-catalog.test.js b/scripts/year3-dart100/test/public-catalog.test.js new file mode 100644 index 0000000..1b244d3 --- /dev/null +++ b/scripts/year3-dart100/test/public-catalog.test.js @@ -0,0 +1,71 @@ +const assert = require('node:assert/strict'); +const test = require('node:test'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); +const vm = require('node:vm'); + +async function runAudit(pages, expected, requireListed = '1') { + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'ain-public-test-')); + fs.mkdirSync(path.join(directory, 'evidence')); + const errors = []; + const calls = []; + const processStub = { env: { RUN_ID: 'test', EXPECTED_PATCH_IDS: expected, CATALOG_REQUIRE_LISTED: requireListed } }; + try { + const source = fs.readFileSync(path.join(__dirname, '..', 'verify-public-catalog.js'), 'utf8'); + await vm.runInNewContext(source, { + require, __dirname: path.join(directory, 'harness'), process: processStub, URL, AbortSignal, + console: { log() {}, error(message) { errors.push(message); } }, + fetch: async url => { + calls.push(url.pathname + url.search); + const body = url.pathname === '/api/info' ? { node: { address: 'test' }, ledger: { kind: 'local' } } : pages.shift(); + assert.ok(body, `unexpected request ${url}`); + return { ok: true, text: async () => JSON.stringify(body) }; + }, + }); + const file = path.join(directory, 'evidence/test/result.json'); + return { code: processStub.exitCode || 0, errors, calls, result: fs.existsSync(file) ? JSON.parse(fs.readFileSync(file)) : null }; + } finally { + fs.rmSync(directory, { recursive: true, force: true }); + } +} + +test('empty public catalog cannot pass without expected IDs', async () => { + const result = await runAudit([{ total: 0, items: [] }], ''); + assert.equal(result.code, 1); + assert.equal(result.result.pass, false); +}); + +test('all 100 expected LISTED patches are checked across pages', async () => { + const items = Array.from({ length: 100 }, (_, index) => ({ anchor: { id: `patch-${index}` }, status: 'LISTED' })); + const result = await runAudit([{ total: 100, items: items.slice(0, 50) }, { total: 100, items: items.slice(50) }], items.map(item => item.anchor.id).join(',')); + assert.equal(result.code, 0); + assert.equal(result.result.matches.length, 100); + assert.equal(result.result.returnedItems, 100); + assert.ok(result.calls.includes('/api/catalog?limit=200&offset=50')); +}); + +test('ANNOUNCED is not LISTED', async () => { + const result = await runAudit([{ total: 1, items: [{ anchor: { id: 'patch' }, status: 'ANNOUNCED' }] }], 'patch'); + assert.equal(result.code, 1); + assert.equal(result.result.matches[0].found, true); + assert.equal(result.result.catalogPresenceComplete, true); + assert.equal(result.result.verificationComplete, false); +}); + +test('presence-only observation reports ANNOUNCED without pretending independent verification passed', async () => { + const result = await runAudit([{ total: 1, items: [{ anchor: { id: 'patch' }, status: 'ANNOUNCED' }] }], 'patch', '0'); + assert.equal(result.code, 0); + assert.equal(result.result.catalogPresenceComplete, true); + assert.equal(result.result.visibleCount, 1); + assert.equal(result.result.listedCount, 0); + assert.equal(result.result.verificationComplete, false); +}); + +test('duplicate pagination IDs fail closed', async () => { + const item = { anchor: { id: 'patch' }, status: 'LISTED' }; + const result = await runAudit([{ total: 2, items: [item] }, { total: 2, items: [item] }], 'patch'); + assert.equal(result.code, 1); + assert.match(result.errors[0], /duplicate/); + assert.equal(result.result, null); +}); diff --git a/scripts/year3-dart100/test/public-proxy.test.js b/scripts/year3-dart100/test/public-proxy.test.js new file mode 100644 index 0000000..f49c31a --- /dev/null +++ b/scripts/year3-dart100/test/public-proxy.test.js @@ -0,0 +1,62 @@ +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const http = require('node:http'); +const { publicRequestAllowed, publicHeaders, createPublicProxy } = require('../ainize-public-proxy'); + +test('only marketplace read/download and native P2P writes are exposed', () => { + for (const target of ['/api/info', '/api/catalog?limit=100', '/api/patches/taught-1/quote', '/x402/patch/taught-1', '/p2p/blob/' + 'a'.repeat(64)]) assert.equal(publicRequestAllowed('GET', target), true); + for (const target of ['/p2p/hello', '/p2p/records']) assert.equal(publicRequestAllowed('POST', target), true); + for (const target of ['/api/auth/setup', '/api/teach/datasets', '/api/me/patches', '/api/chat', '/api/runtime/apply']) { + for (const method of ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS']) assert.equal(publicRequestAllowed(method, target), false); + } +}); + +test('encoded paths, dot segments, authority URLs and non-read methods cannot bypass the allowlist', () => { + for (const target of ['//evil.invalid/api/info', 'http://evil.invalid/api/info', '/api/me/../info', '/api/%69nfo', '/api/info%2f..', '/api\\info', '/api/info\r\nX: 1']) assert.equal(publicRequestAllowed('GET', target), false); + assert.equal(publicRequestAllowed('DELETE', '/api/catalog'), false); + assert.equal(publicRequestAllowed('POST', '/api/info'), false); +}); + +test('operator credentials and spoofed forwarding headers are not sent upstream', () => { + const result = publicHeaders({ authorization: 'Bearer fixture', cookie: 'session=fixture', connection: 'x-remove', 'x-remove': 'drop', 'x-forwarded-for': '127.0.0.1', 'x-ainize-auth': 'signed-fixture', 'x-payment': 'payment-fixture' }); + assert.equal(result.authorization, undefined); + assert.equal(result.cookie, undefined); + assert.equal(result['x-forwarded-for'], undefined); + assert.equal(result['x-remove'], undefined); + assert.equal(result['x-ainize-auth'], 'signed-fixture'); + assert.equal(result['x-payment'], 'payment-fixture'); +}); + +test('real HTTP forwarding preserves bytes and gates mutation, credentials and body size', async context => { + const seen = []; + const upstream = http.createServer((request, response) => { + seen.push({ method: request.method, url: request.url, headers: request.headers }); + const chunks = []; + request.on('data', chunk => chunks.push(chunk)); + request.on('end', () => { + response.writeHead(200, { 'content-type': 'application/octet-stream', 'set-cookie': 'operator=fixture' }); + response.end(request.method === 'POST' ? Buffer.concat(chunks) : Buffer.from([0, 1, 255])); + }); + }); + await new Promise(resolve => upstream.listen(0, '127.0.0.1', resolve)); + const proxy = createPublicProxy({ upstreamPort: upstream.address().port, bodyLimit: 64 }); + await new Promise(resolve => proxy.listen(0, '127.0.0.1', resolve)); + context.after(async () => { + proxy.closeAllConnections(); upstream.closeAllConnections(); + await Promise.all([new Promise(resolve => proxy.close(resolve)), new Promise(resolve => upstream.close(resolve))]); + }); + const origin = `http://127.0.0.1:${proxy.address().port}`; + const response = await fetch(`${origin}/p2p/blob/${'a'.repeat(64)}`, { headers: { authorization: 'Bearer fixture', 'x-ainize-auth': 'signed-fixture' } }); + assert.deepEqual(Buffer.from(await response.arrayBuffer()), Buffer.from([0, 1, 255])); + assert.equal(response.headers.get('set-cookie'), null); + assert.equal(seen[0].headers.authorization, undefined); + assert.equal(seen[0].headers['x-ainize-auth'], 'signed-fixture'); + const denied = await fetch(`${origin}/api/teach/datasets`, { method: 'POST', body: 'data' }); + assert.equal(denied.status, 403); + const oversized = await fetch(`${origin}/p2p/records`, { method: 'POST', body: 'x'.repeat(65) }); + assert.equal(oversized.status, 413); + assert.equal(seen.length, 1); + const posted = await fetch(`${origin}/p2p/records`, { method: 'POST', body: '{"records":[]}' }); + assert.equal(await posted.text(), '{"records":[]}'); + assert.equal(seen.length, 2); +}); diff --git a/scripts/year3-dart100/verify-public-catalog.js b/scripts/year3-dart100/verify-public-catalog.js new file mode 100644 index 0000000..ee87f42 --- /dev/null +++ b/scripts/year3-dart100/verify-public-catalog.js @@ -0,0 +1,63 @@ +const fs = require('fs'); +const path = require('path'); + +async function main() { + const origin = new URL(process.env.PUBLIC_AINIZE_URL || 'https://www.ainize.ai'); + if (origin.protocol !== 'https:' || origin.username || origin.password) { + throw new Error('public verification requires HTTPS without embedded credentials'); + } + const expected = [...new Set((process.env.EXPECTED_PATCH_IDS || '').split(',').map(value => value.trim()).filter(Boolean))]; + const requireListed = process.env.CATALOG_REQUIRE_LISTED !== '0'; + const runId = process.env.RUN_ID || `public-catalog-${Date.now()}`; + if (!/^[a-zA-Z0-9_-]+$/.test(runId)) throw new Error('invalid RUN_ID'); + const directory = path.resolve(__dirname, '..', 'evidence', runId); + fs.mkdirSync(directory, { recursive: false }); + const responses = {}; + for (const endpoint of ['info', 'catalog']) { + const response = await fetch(new URL(`/api/${endpoint}`, origin), { signal: AbortSignal.timeout(30000) }); + const body = await response.text(); + fs.writeFileSync(path.join(directory, `${endpoint}.json`), body, { flag: 'wx' }); + if (!response.ok) throw new Error(`${endpoint}: HTTP ${response.status}`); + responses[endpoint] = JSON.parse(body); + } + if (!Array.isArray(responses.catalog.items)) throw new Error('catalog.items is not an array'); + const total = responses.catalog.total; + if (!Number.isSafeInteger(total) || total < 0 || total > 100000) throw new Error('invalid catalog total'); + const items = [...responses.catalog.items]; + while (items.length < total) { + const offset = items.length; + const response = await fetch(new URL(`/api/catalog?limit=200&offset=${offset}`, origin), { signal: AbortSignal.timeout(30000) }); + const body = await response.text(); + fs.writeFileSync(path.join(directory, `catalog-offset-${offset}.json`), body, { flag: 'wx' }); + if (!response.ok) throw new Error(`catalog offset ${offset}: HTTP ${response.status}`); + const page = JSON.parse(body); + if (page.total !== total || !Array.isArray(page.items) || page.items.length === 0) throw new Error('catalog changed or pagination stalled; repeat with a new RUN_ID'); + items.push(...page.items); + } + const ids = items.map(item => item.anchor?.id); + if (items.length !== total || ids.some(id => typeof id !== 'string' || !id) || new Set(ids).size !== total) { + throw new Error('catalog has missing or duplicate IDs; repeat with a new RUN_ID'); + } + const matches = expected.map(id => { + const item = items.find(candidate => candidate.anchor.id === id); + return { id, found: Boolean(item), status: item?.status || null, listed: item?.status === 'LISTED' }; + }); + const result = { + checkedAt: new Date().toISOString(), origin: origin.origin, + node: responses.info.node?.address, ledger: responses.info.ledger?.kind, + runtimeAvailable: responses.info.runtime?.available, + catalogTotal: total, returnedItems: items.length, + expected, matches, requireListed, + visibleCount: matches.filter(item => item.found).length, + listedCount: matches.filter(item => item.listed).length, + catalogPresenceComplete: expected.length > 0 && matches.every(item => item.found), + verificationComplete: expected.length > 0 && matches.every(item => item.listed), + pass: expected.length > 0 && matches.every(item => requireListed ? item.listed : item.found), + scope: 'Public catalog presence and LISTED verification are separate observations; not an atomic snapshot, payment, model load, dataset, teach or inference proof.', + }; + fs.writeFileSync(path.join(directory, 'result.json'), JSON.stringify(result, null, 2) + '\n', { flag: 'wx' }); + console.log(JSON.stringify(result, null, 2)); + if (!result.pass) process.exitCode = 1; +} + +main().catch(error => { console.error(error.message); process.exitCode = 1; }); From 1e84ed41d953d5d9e05d1f5610520d2c15516033 Mon Sep 17 00:00:00 2001 From: Haechan Date: Fri, 11 Sep 2026 08:10:46 +0000 Subject: [PATCH 4/5] Document empty explore catalogs and unpublished teaching jobs --- scripts/year3-dart100/README.md | 2 ++ ...1\353\241\235_\354\247\204\353\213\250.md" | 35 +++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100644 "scripts/year3-dart100/docs/Ainize_explore_\353\271\210\353\252\251\353\241\235_\354\247\204\353\213\250.md" diff --git a/scripts/year3-dart100/README.md b/scripts/year3-dart100/README.md index eae642f..e491582 100644 --- a/scripts/year3-dart100/README.md +++ b/scripts/year3-dart100/README.md @@ -43,6 +43,8 @@ Install the JavaScript helpers in the existing `/mnt/newdata/gov/kpi/harness` an The experiment's Ainize publisher now uses the public marketplace's `local` ledger. The independent AIN performance chain remains running, and evidence anchors use ain-js separately. Local DAG/CREDIT records are not AIN transfers or blockchain incentive settlement. Public HTTPS callback/Funnel enablement and the publisher's rights/permanence consent are still awaiting operator input. No knowledge has been published to the public marketplace by these helpers, and no verification policy has been weakened to fill the catalog. +The 2026-09-11 08:04–08:06 UTC recheck confirms both catalogs are empty, including explicitly requested ANNOUNCED/VERIFYING statuses. Four local jobs are READY but all have `publish_status: none` and no published patch ID. This is unfinished `teach publish`, not evidence of a browser-cache bug. The [Korean diagnosis and administrator handoff](docs/Ainize_explore_빈목록_진단.md) separate source/package releases, dataset imports, marketplace publication and reverse-download connectivity. + All observer/proxy tests: ```sh diff --git "a/scripts/year3-dart100/docs/Ainize_explore_\353\271\210\353\252\251\353\241\235_\354\247\204\353\213\250.md" "b/scripts/year3-dart100/docs/Ainize_explore_\353\271\210\353\252\251\353\241\235_\354\247\204\353\213\250.md" new file mode 100644 index 0000000..c83148e --- /dev/null +++ "b/scripts/year3-dart100/docs/Ainize_explore_\353\271\210\353\252\251\353\241\235_\354\247\204\353\213\250.md" @@ -0,0 +1,35 @@ +# Ainize explore 빈 목록 진단 + +## 확인 결과 — 2026-09-11 08:04–08:06 UTC + +**이번 실험에서 `ainize teach publish`를 완료한 지식이 아직 없어서 목록이 비어 있다.** GitHub commit/push/release, npm CLI 배포, HF URL 가져오기와 Ainize 마켓플레이스 발행은 서로 다른 작업이다. 이를 같은 “publish 완료”로 표현하지 않는다. + +- 공개 `https://www.ainize.ai/api/catalog`: `total: 0, items: []`. +- LISTED뿐 아니라 ANNOUNCED/VERIFYING/CHALLENGED/SUPERSEDED/REJECTED를 명시해 조회해도 0건이다. 단순 브라우저 새로고침이나 검증 필터 해제로 해결될 상태가 아니다. +- 실험 노드 `http://localhost:3410/api/catalog`도 0건이다. +- 08:06의 로컬 teach 조회에는 READY job 4개가 있지만 **모두 `publish_status: none`, `patch_id: null`**이다. 이들은 private draft이지 공개된 지식이 아니다. READY 자체가 전 문항 정답이나 구매·적용 성공을 뜻하지 않는다. +- HF URL 연동 100개·780행은 완료되었지만, 이를 공개 지식 100개 또는 실제 학습·추론 100개 완료로 세지 않는다. + +공개 API 원문과 로컬 상태 요약은 `kpi/evidence/public_catalog_explore_recheck_20260911T0804/`에 보존했다. 조회 시각이 다르므로 하나의 원자적 스냅샷이라고 주장하지 않는다. + +## 해결된 문제와 남은 작업 + +1. 공개 노드와 실험 노드의 원장 불일치는 **07:39에 해결**했다. 실험 Ainize의 `ledger.kind`만 `ain → local`로 바꿨고, 같은 identity/job을 보존했다. 10개 AIN 성능시험 노드와 GPU 컨테이너는 그대로다. local CREDIT 기록을 AIN 정산이라고 부르지 않는다. +2. **실제 teach 발행은 미실행**이다. CLI가 요구하는 영구 공개·배포권 확인을 게시자에게 요청한 상태이며 두 동의 플래그를 임의로 참으로 만들지 않는다. +3. 판매자 endpoint가 아직 `http://localhost:3410`이다. 원격 서버에서 이 주소는 이 실험 머신을 가리키지 않는다. 따라서 구매자·검증자의 역방향 파일 접근을 입증하지 못했다. 이것은 다운로드/검증을 위한 별도 연결 문제이며 “HTTPS가 없으면 ANNOUNCED 메타데이터도 절대 표시되지 않는다”는 뜻은 아니다. +4. 공개용 제한 프록시는 이 머신의 `127.0.0.1:3412`에서 준비되었지만 Tailscale Funnel은 아직 활성화되지 않았다. `tailscale funnel status`는 `No serve config`이다. 외부 연결이나 공개 배포가 완료되었다고 표시하지 않는다. + +동의와 연결이 준비되면 기존 READY job을 중복 학습하지 않고 다음 실제 명령을 사용한다. + +```sh +ainize teach publish --name "" --consent-permanent --consent-rights +``` + +이후 반환된 실제 knowledge ID로 공개 카탈로그 존재 여부를 검사한다. ANNOUNCED 목록 노출, LISTED 독립 검증, 실제 use/chat의 다운로드·모델 적용·추론은 각각 따로 확인한다. 소스 릴리스가 성공했다는 이유로 이 단계를 통과 처리하지 않는다. + +## 관리자에게 전달할 요청 + +> DART 실험 참여 노드는 `0x20A4e266da261F187613efcb90b1eB131BC381a1`이며 공개 마켓플레이스와 같은 local 원장입니다. 이 노드의 제한 프록시 `127.0.0.1:3412`에 외부 검증자·구매자가 접근할 HTTPS 연결 경로가 필요합니다. 이 머신에서 Tailscale Funnel을 허용하거나, 관리하시는 HTTPS reverse proxy와 이 머신을 연결할 터널의 접속 호스트·사용자·전달 포트·최종 공개 URL을 알려 주세요. 인증 수단은 보호된 파일/서버 설정으로 제공하고 비밀키·토큰은 채팅에 보내지 말아 주세요. 공개 서버의 `localhost:3410`으로 연결하면 이 머신에 도달하지 않습니다. 연결 후 /api/info의 노드 주소와 실제 다운로드 응답을 양쪽에서 확인하겠습니다. + +공개 서버의 원장을 재초기화하거나 원격 teach 정책을 반드시 켜 달라는 요청이 아니다. 자체 노드에서 학습·발행하는 흐름을 유지한다. LISTED까지 검증하려면 이후 호환 모델을 가진 독립 verifier들의 실측 결과도 필요하지만, 단순 목록 노출을 확인하는 조건에 이를 몰래 추가하지 않는다. + From ee90062727c1367bffd5396c9c075ed826deb7c1 Mon Sep 17 00:00:00 2001 From: Haechan Date: Fri, 11 Sep 2026 08:25:48 +0000 Subject: [PATCH 5/5] Record native P2P publication and reproduce missing-body Live test failure --- scripts/year3-dart100/README.md | 4 ++++ ...250\352\263\204_HF\354\227\260\353\217\231.md" | 15 +++++++++++++++ ...2\251\353\241\235_\354\247\204\353\213\250.md" | 13 +++++++++++-- 3 files changed, 30 insertions(+), 2 deletions(-) diff --git a/scripts/year3-dart100/README.md b/scripts/year3-dart100/README.md index e491582..d61b86b 100644 --- a/scripts/year3-dart100/README.md +++ b/scripts/year3-dart100/README.md @@ -45,6 +45,10 @@ The experiment's Ainize publisher now uses the public marketplace's `local` ledg The 2026-09-11 08:04–08:06 UTC recheck confirms both catalogs are empty, including explicitly requested ANNOUNCED/VERIFYING statuses. Four local jobs are READY but all have `publish_status: none` and no published patch ID. This is unfinished `teach publish`, not evidence of a browser-cache bug. The [Korean diagnosis and administrator handoff](docs/Ainize_explore_빈목록_진단.md) separate source/package releases, dataset imports, marketplace publication and reverse-download connectivity. +**08:20 UTC update:** the publisher explicitly confirmed redistribution rights and permanent publication, and required native P2P instead of a Funnel/proxy prerequisite. Two existing READY jobs were then published as `taught-ainize-teach-first-20260-855df1` and `taught-ainize-lifecycle100-2026-cf9a6f`. Both appear in the public catalog and a fresh browser's explore page as ANNOUNCED, with0/2 independent verifiers. Matching complete signed anchors were compared on both nodes and validated with core `LocalLedger.validate`. A third publication, the telephone dataset, was refused by the PII gate and remains unpublished; no automatic private fallback was used. + +**Live test still fails:** the public model is available, but `has_body=false` and `/api/chat/patches` reports `not_held`. An actual public `POST /api/chat` returned HTTP409, “this node does not hold the patch body”. Native P2P currently pushes records but pulls NPZ bodies from peer endpoints; the sender's localhost endpoint cannot serve a different host. No NAT traversal or outbound blob-push support is claimed here. Existing public P2P receive/relay capability and deployment details were requested from the administrator. The pending Funnel CLI was terminated without creating a tunnel. Public source release, catalog visibility, body replication and successful Live test remain distinct outcomes. + All observer/proxy tests: ```sh diff --git "a/scripts/year3-dart100/docs/Ainize_0-3\353\213\250\352\263\204_HF\354\227\260\353\217\231.md" "b/scripts/year3-dart100/docs/Ainize_0-3\353\213\250\352\263\204_HF\354\227\260\353\217\231.md" index fe56e27..2801df3 100644 --- "a/scripts/year3-dart100/docs/Ainize_0-3\353\213\250\352\263\204_HF\354\227\260\353\217\231.md" +++ "b/scripts/year3-dart100/docs/Ainize_0-3\353\213\250\352\263\204_HF\354\227\260\353\217\231.md" @@ -1,5 +1,20 @@ # Ainize 0–3단계와 Hugging Face 연동 +## 최신 진행: 2026-09-11 08:20 UTC — P2P 목록 공개, Live test 미완료 + +사용자가 DART 재배포 권한과 영구 공개에 명시적으로 동의했고, HTTPS 프록시 대신 **네이티브 P2P** 사용을 요구했다. 따라서 Funnel을 필수 조건으로 삼은 이전 안내를 철회했다. 대기 중이던 Funnel 활성화 명령만 종료했으며 공개 터널은 개통하지 않았다. + +- 기존 READY job 두 개를 실제 `teach publish`로 발행했다. 대표자명 `taught-ainize-teach-first-20260-855df1`, 소재지 `taught-ainize-lifecycle100-2026-cf9a6f`이다. 기본 무료 가격0, public dataset, public-source declaration을 사용하고 원래 개인정보/서명/검증 게이트를 그대로 통과했다. +- 공개 카탈로그 **2건·ANNOUNCED·독립 검증0/2**를 확인했다. 새 브라우저 컨텍스트의 실제 `/explore` 화면에도 두 카드가 보이며 스크린샷을 저장했다. `/p2p/records`의 로컬/공개 anchor 본문·해시가 일치한다. 소스 릴리스나 HF 게시를 목록 공개로 대신 보고한 것이 아니다. +- 대표전화 job 발행은 `dataset_pii`(phone, 8행)로 거절됐다. 사용자 권한 확인만으로 이 게이트를 해제하거나 private으로 자동 전환하지 않았다. 실패를 보존하고 미발행으로 둔다. +- **Live test는 아직 실패한다.** 공개 `/api/chat/patches`에서 두 항목 모두 `reason: not_held`, 테스트 가능 items0이다. 실제 공개 `POST /api/chat`도 HTTP409 `this node does not hold the patch body`로 재현했다. 공개 모델은 available=true이지만 본문 `has_body=false`, dataset_held=false다. +- 현재 네이티브 P2P는 서명된 anchor를 push하지만 NPZ 본문은 `GET /p2p/blob/:sha`로 pull한다. peer의 `localhost:3410` 주소가 원격에서 이 머신을 가리키지 않아 본문을 가져오지 못한다. 자동 NAT traversal/본문 push가 이미 있다고 주장하지 않는다. 관리자에게 기존 outbound P2P 본문 수신·릴레이 기능의 프로토콜/주소/배포 버전을 문의했다. 프록시 개통을 다시 요구하거나 LISTED/라이선스 검사를 무력화하지 않는다. + +증빙: `ainize_p2p_publish_first_20260911/`, `ainize_p2p_publish_second_20260911/`, 실패 `ainize_p2p_publish_third_20260911/`, `public_catalog_two_p2p_20260911/`, `public_explore_browser_20260911/`, `ainize_p2p_publish_20260911/`. 모두 `kpi/evidence/` 아래에 있다. **목록 공개2건은 본문 복제·Live test 성공·100개 학습/추론·유상 구매를 뜻하지 않는다.** + +아래07:47 진행과 관리자 요청은 이후 정정된 이전 관측이다. + + ## 최신 진행: 2026-09-11 07:47 UTC - HF native CLI 가져오기가 **100/100 데이터셋·780행** 완료되었다. 모든 원문 로그 SHA를 재검증했고 기존 Ainize ID/정규 해시가 같으며 새 학습 job은 만들지 않았다. `hf_native_dart100_20260911`, exit0이다. diff --git "a/scripts/year3-dart100/docs/Ainize_explore_\353\271\210\353\252\251\353\241\235_\354\247\204\353\213\250.md" "b/scripts/year3-dart100/docs/Ainize_explore_\353\271\210\353\252\251\353\241\235_\354\247\204\353\213\250.md" index c83148e..c27341e 100644 --- "a/scripts/year3-dart100/docs/Ainize_explore_\353\271\210\353\252\251\353\241\235_\354\247\204\353\213\250.md" +++ "b/scripts/year3-dart100/docs/Ainize_explore_\353\271\210\353\252\251\353\241\235_\354\247\204\353\213\250.md" @@ -1,5 +1,13 @@ # Ainize explore 빈 목록 진단 +## 후속 결과 — 2026-09-11 08:20 UTC + +사용자의 배포권·영구 공개 동의 후 기존 READY job 두 개를 실제로 발행했다. 네이티브 P2P anchor broadcast로 공개 카탈로그에 대표자명/소재지 **2건이 ANNOUNCED**로 나타났으며, 새 브라우저의 explore 화면에서도 두 카드를 확인했다. 별도 HTTPS 프록시는 쓰지 않았다. 대표전화 데이터셋은 PII(phone) 검사에 막혀 미발행이고 이를 우회하지 않았다. + +**Live test는 미완료다.** 공개 모델은 available=true이나 두 항목의 본문이 has_body=false, dataset_held=false다. 공개 chat picker는 items0/elsewhere2(reason=not_held)이고 실제 POST /api/chat는 HTTP409 `this node does not hold the patch body`를 반환한다. 현재 P2P는 anchor push와 본문 GET/pull을 구분하며, 판매자 localhost:3410에서는 원격이 파일을 가져올 수 없다. 자동 NAT traversal/본문 push나 공개 서버 적용까지 완료했다고 주장하지 않는다. 관리자에게 기존 outbound P2P 본문 수신·릴레이 기능의 프로토콜/주소/배포 버전을 문의했다. + +첫 지식: https://www.ainize.ai/0x20A4e266da261F187613efcb90b1eB131BC381a1/taught-ainize-teach-first-20260-855df1 . 둘째 ID: `taught-ainize-lifecycle100-2026-cf9a6f`. 독립 검증은 모두0/2다. 공개 원문·HTTP409·P2P 기록은 `kpi/evidence/ainize_p2p_publish_20260911/`, 화면은 `public_explore_browser_20260911/`에 있다. 아래 빈 목록/동의 대기는08:04–08:06 당시 기록이며 현재 상태로 읽지 않는다. + ## 확인 결과 — 2026-09-11 08:04–08:06 UTC **이번 실험에서 `ainize teach publish`를 완료한 지식이 아직 없어서 목록이 비어 있다.** GitHub commit/push/release, npm CLI 배포, HF URL 가져오기와 Ainize 마켓플레이스 발행은 서로 다른 작업이다. 이를 같은 “publish 완료”로 표현하지 않는다. @@ -27,9 +35,10 @@ ainize teach publish --name "" --consent 이후 반환된 실제 knowledge ID로 공개 카탈로그 존재 여부를 검사한다. ANNOUNCED 목록 노출, LISTED 독립 검증, 실제 use/chat의 다운로드·모델 적용·추론은 각각 따로 확인한다. 소스 릴리스가 성공했다는 이유로 이 단계를 통과 처리하지 않는다. -## 관리자에게 전달할 요청 +## 최초 관리자 요청 — 철회/정정됨 + +사용자가 네이티브 P2P를 요구했으므로 아래 HTTPS/Funnel 요청은 현재 실행 조건이 아니다. 활성화 대기 명령만 종료했으며 터널을 만들지 않았다. 현재 요청은 이 문서의 최신 결과에 적은 **outbound P2P 본문 수신·릴레이 지원 여부/배포 정보**다. 사용자 배포 동의는 이미 받았다. > DART 실험 참여 노드는 `0x20A4e266da261F187613efcb90b1eB131BC381a1`이며 공개 마켓플레이스와 같은 local 원장입니다. 이 노드의 제한 프록시 `127.0.0.1:3412`에 외부 검증자·구매자가 접근할 HTTPS 연결 경로가 필요합니다. 이 머신에서 Tailscale Funnel을 허용하거나, 관리하시는 HTTPS reverse proxy와 이 머신을 연결할 터널의 접속 호스트·사용자·전달 포트·최종 공개 URL을 알려 주세요. 인증 수단은 보호된 파일/서버 설정으로 제공하고 비밀키·토큰은 채팅에 보내지 말아 주세요. 공개 서버의 `localhost:3410`으로 연결하면 이 머신에 도달하지 않습니다. 연결 후 /api/info의 노드 주소와 실제 다운로드 응답을 양쪽에서 확인하겠습니다. 공개 서버의 원장을 재초기화하거나 원격 teach 정책을 반드시 켜 달라는 요청이 아니다. 자체 노드에서 학습·발행하는 흐름을 유지한다. LISTED까지 검증하려면 이후 호환 모델을 가진 독립 verifier들의 실측 결과도 필요하지만, 단순 목록 노출을 확인하는 조건에 이를 몰래 추가하지 않는다. -