Feat/jarvis 3d scene generation#1916
Open
josemguerrajmg-oss wants to merge 5 commits into
Open
Conversation
Adds a new 'scene' mode to the Jarvis HUD where users can describe a 3D
composition in natural language and see it rendered live in Three.js. The
scene generation pipeline calls a local Ollama instance (llama3.2) so
no cloud API key is required for this feature.
Changes:
- shared/types/jarvis.ts: add 'scene' JarvisMode and 'create_scene' command type
- command-parser.ts: detect create/generate/make/build + 3D keywords → create_scene intent
- Jarvis.tsx:
- SceneDescriptor type + callLocalLLMForScene (Ollama /api/chat)
- buildSceneFromDescriptor: maps JSON objects to Three.js meshes (sphere/box/torus/
cylinder/cone/icosahedron/ring/plane with full material support)
- JarvisViewport: sceneGroup rendered procedurally, gentle rotation + float animation,
camera positioned at 6.5z for composed scenes
- executeCommand: create_scene branch routes to Ollama before standard Anthropic path
- Context-aware chips for scene mode: Create new, Brand 3D, Product 3D
- Mode badge, stateChips, and commandChips updated for 'scene'
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switches 3D scene generation from Ollama (required separate install) to the API key already stored in the app keychain. Users get scene rendering as soon as they have an API key configured — no additional setup required. Also widens the trigger regex to catch more natural phrasings like "design a 3D brand mark" or "3D render of a product". Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…overy Scene generation now calls local Ollama instead of the cloud API. Automatically picks the best available model by querying /api/tags and ranking against a preferred list (qwen2.5 > llama3.2 > llama3 > mistral > phi4 > gemma3 > …). Falls back gracefully with actionable error messages if Ollama isn't running or has no models installed. No API key required for 3D scene generation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause of broken scene generation: small local models (llama3.2 etc.) regularly emit malformed JSON — e.g. a mismatched quote like "rotation':[…] — which crashed JSON.parse and produced no scene. Verified against real Ollama. Fixes: - Use Ollama "format":"json" constrained decoding — guarantees parseable JSON output every time. This is the core fix. - Filter out embedding-only models (nomic-embed-text, *-minilm, bert families) so the fallback model picker never selects a model that can't chat. - normalizeDescriptor(): coerce and clamp every field from the model so a single bad value can't break the render — type aliases (cube->box, orb->sphere), numeric range clamps, hex/CSS color validation, and opacity<->transparent coupling. - Defensive JSON repair (smart quotes, single quotes, trailing commas) as a second line of defense. - Wrap fetches in try/catch to surface a clean "ollama_down" instead of an unhandled rejection. - Defer the scene-mode switch until meshes are ready so the user keeps seeing their current view during the multi-second generation instead of an empty void; actionable error messages for each failure mode. Verified: tsc --noEmit clean, vite build succeeds (renderer bundle emitted), and end-to-end generation against local Ollama renders valid scenes for 4 varied prompts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The importer was hardcoded to GLTFLoader with an accept=".glb,.gltf" file picker, so STL (and OBJ/FBX/PLY/DAE/3MF) files silently failed. Adds a format-dispatching loader: - glb/gltf -> GLTFLoader - stl -> STLLoader (geometry-only; wrapped in a metallic standard material) - obj -> OBJLoader - fbx -> FBXLoader - ply -> PLYLoader (geometry-only; enables vertexColors when present) - dae -> ColladaLoader - 3mf -> ThreeMFLoader Each loader is dynamically imported, so it builds into its own lazy chunk and only downloads when that format is actually picked — the main bundle stays lean. Other changes: - File picker now accepts all supported extensions. - Unsupported extensions get a clear "Supported: ..." message instead of a silent failure. - Loads now report progress in the transcript: "Loading file.stl..." -> "STL model loaded." or an actionable error, via an onModelResult callback. - Loading a model auto-switches to object mode (where the model renders), centers and scales it to fit, and disposes the previous model's geometry/materials to avoid leaks. Verified: tsc clean; vite build emits 7 separate loader chunks; and STL/OBJ/PLY parsers confirmed to parse real geometry in a node smoke test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
1. AI 3D scene generation (local Ollama)
A new Scene mode in the Jarvis HUD: describe a 3D composition in plain
language ("a glowing orb surrounded by orbiting violet rings") and it renders
live in Three.js — no cloud API key required.
format: "json"(constrained decoding) so smallmodels reliably return parseable scene JSON
("Ollama not running. Start it with: ollama serve")
2. Universal 3D model import
The importer was GLB/GLTF-only, so STL and other formats silently failed.
Now supports GLB, GLTF, STL, OBJ, FBX, PLY, DAE, 3MF.
vertex colors
Verification
tsc --noEmitcleanvite buildclean (7 separate loader chunks emitted)llama3.2) — 4 prompts, all valid