chore: fallback v2

[sarahxsanders]

our basic language skills are flaky at best - sometimes they work, but they never produce as good of an outcome as the framework-specific skills

the fallback skills are too constrained. strict detection requirements were causing it to do weird things with projects we can't predict the shape of

sister PR: PostHog/context-mill#123

changes

Vanilla HTML/CSS/JS:

• Projects with HTML files but no package.json now auto-detect instead of requiring manual selection
• Agent gets told whether it's vanilla (use CDN script tag) or bundled (use npm install)
• HTML entry point is passed to the agent so it knows where to add the snippet

Node.js:

• New utils.ts with shared detection helpers
• Improved context gathering for plain Node.js projects

Python:

• Detects entry point (main.py, app.py, etc.) and passes it to the agent
• Detects project layout (src layout, lib layout, flat)
• Scans dependencies for patterns (CLI, background worker, database, async framework, data/ML) so the agent understands what kind of Python project it's working with

Ruby:

• Improved detection and context gathering for plain Ruby projects

detection order in constants.ts is unchanged. frameworks are checked first, fallbacks last

Testing

A/B tested all four fallback languages across three configurations:

1. Published – npx @posthog/wizard@latest (current production)
2. This branch + example apps – our wizard changes paired with context-mill, example apps still connected
3. This branch, no example apps – same wizard changes, but example apps disconnected from fallback skills (docs-only)

Each test ran on minimal, no-framework apps: plain http.server, plain http.createServer, vanilla HTML, plain WEBrick

Vanilla HTML/CSS/JS

	Published	+ example apps	Docs-only (no examples)
Auto-detected?	No — manual selection	Yes	Yes
API key	Hardcoded in HTML	Placeholders	Placeholders
Events	signup_clicked	signup_clicked	signup_clicked

Python

	Published	+ example apps	Docs-only (no examples)
API key	Env vars via dotenv	Env vars via os.environ	Env vars via os.environ
Events	signup_form_viewed, signup_completed	page_viewed, user_signed_up	signup_form_viewed, user_signed_up
User ID	Random UUID	SHA256 hash	SHA256 hash
Identify	No	No	Yes — posthog.set()
Error tracking	Autocapture only	No	capture_exception + autocapture
Graceful shutdown	Yes (atexit)	No	Yes (atexit)

Node.js

	Published	+ example apps	Docs-only (no examples)
API key	Env vars (no dotenv)	Env vars via dotenv	Env vars via dotenv
Events	user signed up (spaces!)	page_viewed, user_signed_up	signup_page_viewed, user_signed_up
User ID	Raw email	SHA256 hash	crypto.randomUUID()
Identify	Yes	No	Yes
Error tracking	captureException	No	captureException + autocapture
Graceful shutdown	Yes (SIGTERM + SIGINT)	Yes (SIGTERM)	Yes (SIGINT)

Ruby

	Published	+ example apps	Docs-only (no examples)
API key	Env vars via dotenv	Env vars via dotenv	Env vars
User ID	IP address!	SecureRandom.uuid	SecureRandom.uuid
Identify	Yes — hardcoded plan: 'free'	No	Yes — real properties
Error tracking	No	Error tracking	capture_exception in rescue blocks
Graceful shutdown	Yes	Yes	Yes (at_exit + trap)

with example apps connected, the agent was copying patterns from the example but skipping identify, error tracking, and graceful shutdown. it was too constrained.

without them, the agent leaned on the SDK docs and our fallback-description.md guidance – and consistently produced more complete integrations

seems fallback skills just need some docs, general guidance, and some freedom

The big wins

• Vanilla auto-detection: no longer requires manual framework selection
• API key safety: no more hardcoded keys in vanilla HTML projects
• User ID handling: no more raw emails (Node) or IP addresses (Ruby) as distinct IDs
• Identify + error tracking: consistently present across all languages now (previously hit or miss)
• Event naming: clean snake_case names, no spaces

[github-actions]

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

• /wizard-ci all

Test all apps in a directory:

• /wizard-ci basic-integration
• /wizard-ci misc
• /wizard-ci revenue

Test an individual app:

• /wizard-ci basic-integration/android
• /wizard-ci basic-integration/angular
• /wizard-ci basic-integration/astro

Show more apps

• /wizard-ci basic-integration/django
• /wizard-ci basic-integration/fastapi
• /wizard-ci basic-integration/flask
• /wizard-ci basic-integration/javascript-node
• /wizard-ci basic-integration/javascript-web
• /wizard-ci basic-integration/laravel
• /wizard-ci basic-integration/next-js
• /wizard-ci basic-integration/nuxt
• /wizard-ci basic-integration/python
• /wizard-ci basic-integration/rails
• /wizard-ci basic-integration/react-native
• /wizard-ci basic-integration/react-router
• /wizard-ci basic-integration/sveltekit
• /wizard-ci basic-integration/swift
• /wizard-ci basic-integration/tanstack-router
• /wizard-ci basic-integration/tanstack-start
• /wizard-ci basic-integration/vue
• /wizard-ci misc/quack-quack
• /wizard-ci revenue/stripe

Results will be posted here when complete.