diff --git a/content/docs/guides/meta.json b/content/docs/guides/meta.json index 5ce534f32..81b1996cd 100644 --- a/content/docs/guides/meta.json +++ b/content/docs/guides/meta.json @@ -2,6 +2,7 @@ "title": "Guides", "pages": [ "index", + "packages", "metadata", "data-modeling", "common-patterns", diff --git a/content/docs/guides/packages.mdx b/content/docs/guides/packages.mdx new file mode 100644 index 000000000..811801216 --- /dev/null +++ b/content/docs/guides/packages.mdx @@ -0,0 +1,473 @@ +--- +title: Package Overview +description: Complete guide to all ObjectStack packages, services, drivers, plugins, and adapters +--- + +# Package Overview + +ObjectStack is organized into **27 packages** across multiple categories. This guide provides a complete overview of each package, its purpose, and when to use it. + +## Core Packages + +### @objectstack/spec + +**The Constitution** — Protocol schemas, types, and constants for the entire ObjectStack ecosystem. + +- **Purpose**: Zod-first schema definitions for all 15 protocol domains +- **Exports**: Data, UI, System, Automation, AI, API, Identity, Security, Kernel, Cloud, QA, Contracts, Integration, Studio, Shared namespaces +- **When to use**: Import types, schemas, and builder functions (`defineObject`, `defineView`, `defineFlow`, etc.) +- **Documentation**: [Protocol Reference](/docs/references) + +```typescript +import { Data, UI, defineObject, defineView } from '@objectstack/spec'; +``` + +### @objectstack/core + +**The Microkernel** — DI container, plugin manager, and service registry. + +- **Purpose**: ObjectKernel with dependency injection, lifecycle hooks, and event bus +- **Exports**: `ObjectKernel`, `LiteKernel`, `Plugin` interface, service management +- **When to use**: Bootstrap your application, manage plugins and services +- **README**: [View README](/packages/core/README.md) + +```typescript +import { ObjectKernel } from '@objectstack/core'; +const kernel = new ObjectKernel(); +``` + +### @objectstack/runtime + +**Runtime Bootstrap** — DriverPlugin, AppPlugin, and capability contracts. + +- **Purpose**: High-level runtime bootstrap and plugin composition +- **Exports**: Runtime configuration, plugin loaders, capability system +- **When to use**: Use with `defineStack()` for application setup +- **README**: [View README](/packages/runtime/README.md) + +### @objectstack/objectql + +**Data Query Engine** — MongoDB-style queries with SQL execution. + +- **Purpose**: ObjectQL query engine with filters, aggregations, and window functions +- **Exports**: Query parser, filter engine, schema registry +- **When to use**: Advanced query operations, custom data access patterns +- **README**: [View README](/packages/objectql/README.md) + +### @objectstack/metadata + +**Metadata Management** — Loaders, serializers, and overlay system. + +- **Purpose**: Load, validate, and manage metadata from files or runtime +- **Exports**: Metadata loaders, serializers, overlay system, validation +- **When to use**: Dynamic metadata loading, multi-source metadata composition +- **README**: [View README](/packages/metadata/README.md) + +### @objectstack/rest + +**REST API Layer** — Auto-generated REST endpoints from metadata. + +- **Purpose**: Automatic REST API generation based on object definitions +- **Exports**: REST server, route generators, middleware +- **When to use**: Expose ObjectStack data via REST API +- **README**: [View README](/packages/rest/README.md) + +--- + +## Client Packages + +### @objectstack/client + +**Framework-Agnostic SDK** — Universal TypeScript client for ObjectStack. + +- **Purpose**: Type-safe client for ObjectStack REST API with batching and error handling +- **Exports**: `ObjectStackClient`, query builders, error classes +- **When to use**: Any JavaScript/TypeScript application (Node, React, Vue, Angular, etc.) +- **README**: [View README](/packages/client/README.md) + +```typescript +import { ObjectStackClient } from '@objectstack/client'; +const client = new ObjectStackClient({ baseURL: 'https://api.example.com' }); +``` + +### @objectstack/client-react + +**React Hooks & Bindings** — React hooks for ObjectStack. + +- **Purpose**: React hooks for queries, mutations, real-time subscriptions +- **Exports**: `useQuery`, `useMutation`, `useRealtime`, `useAuth`, etc. +- **When to use**: React applications +- **README**: [View README](/packages/client-react/README.md) + +```typescript +import { useQuery, useMutation } from '@objectstack/client-react'; +``` + +--- + +## Data Drivers + +### @objectstack/driver-memory + +**In-Memory Driver** — Ephemeral storage for development and testing. + +- **Purpose**: Fast in-memory data storage with full ObjectQL support +- **When to use**: Development, testing, demos (data is lost on restart) +- **README**: [View README](/packages/plugins/driver-memory/README.md) + +```typescript +import { DriverMemory } from '@objectstack/driver-memory'; +``` + +### @objectstack/driver-sql + +**SQL Driver** — PostgreSQL, MySQL, SQLite support via Knex.js. + +- **Purpose**: Production-ready SQL database support with migrations +- **Supports**: PostgreSQL, MySQL, SQLite, and all Knex-compatible databases +- **When to use**: Traditional relational database deployments +- **README**: [View README](/packages/plugins/driver-sql/README.md) + +```typescript +import { DriverSQL } from '@objectstack/driver-sql'; +const driver = DriverSQL.configure({ + client: 'pg', + connection: { /* PostgreSQL config */ }, +}); +``` + +### @objectstack/driver-turso + +**Turso/libSQL Driver** — Edge-first SQLite with embedded replicas. + +- **Purpose**: Edge-native SQLite with local-first architecture and multi-tenancy +- **Modes**: Remote (edge), Embedded Replica (local-first), Local (dev) +- **When to use**: Vercel Edge, Cloudflare Workers, global low-latency deployments +- **README**: [View README](/packages/plugins/driver-turso/README.md) + +```typescript +import { DriverTurso } from '@objectstack/driver-turso'; +const driver = DriverTurso.configure({ + url: 'libsql://mydb-username.turso.io', + authToken: process.env.TURSO_AUTH_TOKEN, +}); +``` + +--- + +## Platform Services + +All services implement contracts from `@objectstack/spec/contracts` and are kernel-managed singletons. + +### @objectstack/service-ai + +**AI Service** — LLM adapter layer, conversation management, tool registry. + +- **Supports**: OpenAI, Anthropic, Google, custom providers via Vercel AI SDK +- **Features**: Multi-model support, streaming, tool calling, conversation history +- **When to use**: AI-powered features, chatbots, agents, RAG pipelines +- **README**: [View README](/packages/services/service-ai/README.md) + +### @objectstack/service-analytics + +**Analytics Service** — Multi-driver analytics with NativeSQL, ObjectQL, InMemory strategies. + +- **Features**: Aggregations, time series, funnels, dashboards +- **When to use**: Business intelligence, reporting, metrics dashboards +- **README**: [View README](/packages/services/service-analytics/README.md) + +### @objectstack/service-automation + +**Automation Service** — DAG flow execution engine for workflows. + +- **Features**: Autolaunched, screen, and scheduled flows with visual builder support +- **When to use**: Business process automation, approval workflows, scheduled tasks +- **README**: [View README](/packages/services/service-automation/README.md) + +### @objectstack/service-cache + +**Cache Service** — In-memory and Redis caching. + +- **Adapters**: Memory (dev), Redis (production) +- **Features**: TTL, namespaces, pattern matching, statistics +- **When to use**: Performance optimization, reduce database load +- **README**: [View README](/packages/services/service-cache/README.md) + +### @objectstack/service-feed + +**Feed/Chatter Service** — Activity feed with comments, reactions, subscriptions. + +- **Features**: Comments, @mentions, reactions, field change tracking, presence +- **When to use**: Collaboration features, activity streams, social features +- **README**: [View README](/packages/services/service-feed/README.md) + +### @objectstack/service-i18n + +**I18n Service** — Internationalization with file-based locales. + +- **Features**: Multi-language support, interpolation, pluralization, fallback chains +- **When to use**: Multi-language applications, global deployments +- **README**: [View README](/packages/services/service-i18n/README.md) + +### @objectstack/service-job + +**Job Service** — Cron and interval-based job scheduling. + +- **Features**: Cron expressions, intervals, one-time jobs, retry logic, history +- **When to use**: Background tasks, scheduled reports, cleanup jobs +- **README**: [View README](/packages/services/service-job/README.md) + +### @objectstack/service-queue + +**Queue Service** — Job queues with in-memory and BullMQ adapters. + +- **Features**: Priority queues, retry, rate limiting, worker pools, job events +- **When to use**: Async processing, email sending, report generation, webhooks +- **README**: [View README](/packages/services/service-queue/README.md) + +### @objectstack/service-realtime + +**Realtime Service** — WebSocket pub/sub for live updates. + +- **Features**: Channels, presence, broadcasting, typing indicators, cursor tracking +- **When to use**: Real-time dashboards, collaborative editing, live notifications +- **README**: [View README](/packages/services/service-realtime/README.md) + +### @objectstack/service-storage + +**Storage Service** — File storage with local filesystem and S3 adapters. + +- **Features**: Upload, download, signed URLs, multipart uploads, metadata +- **When to use**: File attachments, document management, media storage +- **README**: [View README](/packages/services/service-storage/README.md) + +--- + +## Official Plugins + +### @objectstack/plugin-auth + +**Authentication Plugin** — Better-auth integration with ObjectQL. + +- **Features**: Email/password, OAuth providers, session management, RBAC +- **When to use**: User authentication and authorization +- **README**: [View README](/packages/plugins/plugin-auth/README.md) + +### @objectstack/plugin-security + +**Security Plugin** — RBAC, permissions, field-level security. + +- **Features**: Role-based access control, object/field permissions, row-level security +- **When to use**: Multi-user applications with access control requirements +- **README**: [View README](/packages/plugins/plugin-security/README.md) + +### @objectstack/plugin-audit + +**Audit Plugin** — Compliance audit trail and activity logging. + +- **Features**: CRUD audit logs, field-level changes, security events, compliance reports +- **When to use**: SOC 2, HIPAA, GDPR compliance, security monitoring +- **README**: [View README](/packages/plugins/plugin-audit/README.md) + +### @objectstack/plugin-mcp-server + +**MCP Server Plugin** — Expose ObjectStack via Model Context Protocol. + +- **Features**: AI tools, data resources, prompt templates for Claude, Cursor, Cline +- **When to use**: AI agent integration, MCP-compatible tools +- **README**: [View README](/packages/plugins/plugin-mcp-server/README.md) + +### @objectstack/plugin-hono-server + +**Hono Server Plugin** — HTTP server with Hono framework. + +- **Features**: Lightweight HTTP server, middleware support, edge-compatible +- **When to use**: Serve ObjectStack REST API with Hono +- **README**: [View README](/packages/plugins/plugin-hono-server/README.md) + +### @objectstack/plugin-msw + +**MSW Plugin** — Mock Service Worker for testing. + +- **Features**: Mock HTTP handlers for testing without real server +- **When to use**: Frontend testing, integration tests +- **README**: [View README](/packages/plugins/plugin-msw/README.md) + +### @objectstack/plugin-dev + +**Developer Tools Plugin** — Development-time utilities. + +- **Features**: Metadata validation, schema introspection, debugging tools +- **When to use**: Development and debugging +- **README**: [View README](/packages/plugins/plugin-dev/README.md) + +### @objectstack/plugin-setup + +**Setup Plugin** — First-run setup wizard. + +- **Features**: Initial configuration, sample data, guided setup +- **When to use**: New project initialization +- **README**: [View README](/packages/plugins/plugin-setup/README.md) + +--- + +## Framework Adapters + +ObjectStack integrates with popular web frameworks via adapters. All adapters expose ObjectStack's REST API through the framework's routing system. + +### @objectstack/express + +**Express.js Adapter** — Traditional Node.js web framework. + +- **Use case**: Classic Node.js applications, RESTful APIs +- **README**: [View README](/packages/adapters/express/README.md) + +### @objectstack/fastify + +**Fastify Adapter** — High-performance Node.js framework. + +- **Use case**: Performance-critical applications, microservices +- **README**: [View README](/packages/adapters/fastify/README.md) + +### @objectstack/hono + +**Hono Adapter** — Edge-native web framework. + +- **Use case**: Cloudflare Workers, Vercel Edge, Deno, Bun +- **README**: [View README](/packages/adapters/hono/README.md) + +### @objectstack/nestjs + +**NestJS Adapter** — Enterprise TypeScript framework. + +- **Use case**: Enterprise applications, complex architectures +- **README**: [View README](/packages/adapters/nestjs/README.md) + +### @objectstack/nextjs + +**Next.js Adapter** — React metaframework. + +- **Use case**: Full-stack React applications, App Router, Server Components +- **README**: [View README](/packages/adapters/nextjs/README.md) + +### @objectstack/nuxt + +**Nuxt Adapter** — Vue metaframework. + +- **Use case**: Full-stack Vue applications +- **README**: [View README](/packages/adapters/nuxt/README.md) + +### @objectstack/sveltekit + +**SvelteKit Adapter** — Svelte metaframework. + +- **Use case**: Full-stack Svelte applications +- **README**: [View README](/packages/adapters/sveltekit/README.md) + +--- + +## Developer Tools + +### @objectstack/cli + +**CLI Tool** — Command-line interface for ObjectStack. + +- **Commands**: `serve`, `studio`, `doctor`, `migrate`, `deploy` +- **When to use**: Development, deployment, project management +- **README**: [View README](/packages/cli/README.md) + +```bash +npx @objectstack/cli serve --dev +``` + +### @objectstack/create-objectstack + +**Project Scaffolding** — Create new ObjectStack projects. + +- **Templates**: Minimal, CRM, Todo, E-commerce +- **When to use**: Start a new ObjectStack project +- **README**: [View README](/packages/create-objectstack/README.md) + +```bash +npx create-objectstack my-app +``` + +### @objectstack/vscode-objectstack + +**VS Code Extension** — IDE support for ObjectStack. + +- **Features**: Syntax highlighting, autocomplete, validation for metadata files +- **When to use**: Enhanced development experience in VS Code +- **README**: [View README](/packages/vscode-objectstack/README.md) + +--- + +## Utility Packages + +### @objectstack/types + +**Shared Type Utilities** — Common TypeScript types and utilities. + +- **Exports**: Type helpers, utility types, branded types +- **When to use**: Imported automatically by other packages +- **README**: [View README](/packages/types/README.md) + +--- + +## Package Selection Guide + +### For New Projects + +```typescript +import { defineStack } from '@objectstack/spec'; +import { DriverTurso } from '@objectstack/driver-turso'; // Edge-first +import { ServiceAI } from '@objectstack/service-ai'; // AI features + +const stack = defineStack({ + driver: DriverTurso.configure({ /* ... */ }), + services: [ServiceAI.configure({ /* ... */ })], +}); +``` + +### For Traditional Web Apps + +```typescript +import { DriverSQL } from '@objectstack/driver-sql'; // PostgreSQL/MySQL +import { PluginAuth } from '@objectstack/plugin-auth'; // Authentication +import { ServiceQueue } from '@objectstack/service-queue'; // Background jobs + +const stack = defineStack({ + driver: DriverSQL.configure({ client: 'pg', /* ... */ }), + plugins: [PluginAuth.configure({ /* ... */ })], + services: [ServiceQueue.configure({ /* ... */ })], +}); +``` + +### For Enterprise Applications + +```typescript +import { PluginSecurity } from '@objectstack/plugin-security'; // RBAC +import { PluginAudit } from '@objectstack/plugin-audit'; // Compliance +import { ServiceAnalytics } from '@objectstack/service-analytics'; // BI + +const stack = defineStack({ + driver: DriverSQL.configure({ /* PostgreSQL */ }), + plugins: [ + PluginSecurity.configure({ /* ... */ }), + PluginAudit.configure({ /* ... */ }), + ], + services: [ + ServiceAnalytics.configure({ /* ... */ }), + ServiceI18n.configure({ /* ... */ }), + ], +}); +``` + +--- + +## Next Steps + +- **Architecture Overview**: [Learn the architecture](/docs/getting-started/architecture) +- **Quick Start**: [Build your first app](/docs/getting-started/quick-start) +- **API Reference**: [Explore the API](/docs/references) +- **Examples**: [Browse examples](/docs/getting-started/examples) diff --git a/content/docs/index.mdx b/content/docs/index.mdx index 7010522d5..b92da8c9a 100644 --- a/content/docs/index.mdx +++ b/content/docs/index.mdx @@ -45,6 +45,12 @@ New to ObjectStack? Start here. href="/docs/getting-started/examples" description="Run the Todo, CRM, and BI examples to learn hands-on." /> + } + title="Package Overview" + href="/docs/guides/packages" + description="Complete guide to all 27 packages: core, services, drivers, plugins, and adapters." + /> ## Build diff --git a/packages/plugins/driver-sql/README.md b/packages/plugins/driver-sql/README.md new file mode 100644 index 000000000..c74575724 --- /dev/null +++ b/packages/plugins/driver-sql/README.md @@ -0,0 +1,476 @@ +# @objectstack/driver-sql + +SQL Driver for ObjectStack - Supports PostgreSQL, MySQL, SQLite via Knex.js. + +## Features + +- **Multi-Database Support**: PostgreSQL, MySQL, SQLite, and other Knex-supported databases +- **Query Builder**: Powerful Knex.js query builder integration +- **Migrations**: Database schema migrations with version control +- **Connection Pooling**: Efficient connection management +- **Transactions**: Full ACID transaction support +- **Raw SQL**: Execute raw SQL when needed +- **Type-Safe**: Full TypeScript support with inferred types +- **Production-Ready**: Battle-tested Knex.js under the hood + +## Installation + +```bash +pnpm add @objectstack/driver-sql knex +``` + +### Database-Specific Drivers + +Install the driver for your database: + +```bash +# PostgreSQL +pnpm add pg + +# MySQL +pnpm add mysql2 + +# SQLite +pnpm add better-sqlite3 +``` + +## Basic Usage + +### PostgreSQL + +```typescript +import { defineStack } from '@objectstack/spec'; +import { DriverSQL } from '@objectstack/driver-sql'; + +const stack = defineStack({ + driver: DriverSQL.configure({ + client: 'pg', + connection: { + host: 'localhost', + port: 5432, + user: 'postgres', + password: process.env.DB_PASSWORD, + database: 'myapp', + }, + pool: { + min: 2, + max: 10, + }, + }), +}); +``` + +### MySQL + +```typescript +const stack = defineStack({ + driver: DriverSQL.configure({ + client: 'mysql2', + connection: { + host: 'localhost', + port: 3306, + user: 'root', + password: process.env.DB_PASSWORD, + database: 'myapp', + }, + }), +}); +``` + +### SQLite + +```typescript +const stack = defineStack({ + driver: DriverSQL.configure({ + client: 'better-sqlite3', + connection: { + filename: './data/app.db', + }, + useNullAsDefault: true, + }), +}); +``` + +## Configuration Options + +```typescript +interface SQLDriverConfig { + /** Knex client (pg, mysql2, better-sqlite3, etc.) */ + client: string; + + /** Database connection config */ + connection: { + host?: string; + port?: number; + user?: string; + password?: string; + database?: string; + filename?: string; // For SQLite + }; + + /** Connection pool settings */ + pool?: { + min?: number; + max?: number; + idleTimeoutMillis?: number; + }; + + /** Use NULL as default for unsupported features (SQLite) */ + useNullAsDefault?: boolean; + + /** Enable query debugging */ + debug?: boolean; + + /** Migrations configuration */ + migrations?: { + directory?: string; + tableName?: string; + }; +} +``` + +## Database Operations + +The SQL driver implements the standard ObjectStack driver interface: + +```typescript +import type { IDriver } from '@objectstack/spec'; + +// All standard operations are supported: +// find, findOne, insert, update, delete, count +``` + +### Advanced Queries + +```typescript +// The SQL driver supports all ObjectQL query features: +const results = await kernel.getDriver().find({ + object: 'opportunity', + filters: [ + { field: 'amount', operator: 'gte', value: 10000 }, + { field: 'stage', operator: 'in', value: ['proposal', 'negotiation'] }, + ], + sort: [{ field: 'amount', direction: 'desc' }], + limit: 100, + offset: 0, +}); +``` + +## Migrations + +### Creating Migrations + +```typescript +// migrations/001_create_users.ts +export async function up(knex) { + await knex.schema.createTable('objectstack_user', (table) => { + table.string('id').primary(); + table.string('name').notNullable(); + table.string('email').notNullable().unique(); + table.timestamps(true, true); + }); +} + +export async function down(knex) { + await knex.schema.dropTable('objectstack_user'); +} +``` + +### Running Migrations + +```bash +# Run all pending migrations +npx knex migrate:latest + +# Rollback last migration +npx knex migrate:rollback + +# Check migration status +npx knex migrate:status +``` + +### Migration Configuration + +Create `knexfile.js` in your project root: + +```javascript +module.exports = { + development: { + client: 'pg', + connection: { + host: 'localhost', + user: 'postgres', + password: process.env.DB_PASSWORD, + database: 'myapp_dev', + }, + migrations: { + directory: './migrations', + tableName: 'objectstack_migrations', + }, + }, + production: { + client: 'pg', + connection: process.env.DATABASE_URL, + pool: { + min: 2, + max: 10, + }, + migrations: { + directory: './migrations', + tableName: 'objectstack_migrations', + }, + }, +}; +``` + +## Transactions + +```typescript +const driver = kernel.getDriver(); + +await driver.transaction(async (trx) => { + // All operations within this callback use the same transaction + const account = await trx.insert({ + object: 'account', + data: { name: 'Acme Corp' }, + }); + + await trx.insert({ + object: 'contact', + data: { + name: 'John Doe', + account_id: account.id, + }, + }); + + // If an error is thrown, all changes are rolled back + // If successful, changes are committed +}); +``` + +## Raw SQL Queries + +When ObjectQL isn't sufficient, execute raw SQL: + +```typescript +const driver = kernel.getDriver(); + +// Raw query +const results = await driver.raw(` + SELECT + c.name, + COUNT(o.id) as opportunity_count, + SUM(o.amount) as total_revenue + FROM objectstack_account c + LEFT JOIN objectstack_opportunity o ON o.account_id = c.id + WHERE o.stage = 'closed_won' + GROUP BY c.id, c.name + ORDER BY total_revenue DESC + LIMIT 10 +`); + +// Raw query with parameters (prevent SQL injection) +const results = await driver.raw( + 'SELECT * FROM objectstack_user WHERE email = ?', + ['user@example.com'] +); +``` + +## Database-Specific Features + +### PostgreSQL Features + +```typescript +// Use PostgreSQL-specific features +const results = await driver.raw(` + SELECT * FROM objectstack_opportunity + WHERE data @> '{"industry": "Technology"}'::jsonb +`); + +// Full-text search +const results = await driver.raw(` + SELECT * FROM objectstack_article + WHERE to_tsvector('english', title || ' ' || body) @@ to_tsquery('objectstack') +`); +``` + +### MySQL Features + +```typescript +// Use MySQL-specific features +const results = await driver.raw(` + SELECT * FROM objectstack_product + WHERE MATCH(name, description) AGAINST ('widget' IN NATURAL LANGUAGE MODE) +`); +``` + +## Connection Management + +```typescript +// Get underlying Knex instance +const knex = driver.getKnex(); + +// Check connection +await driver.checkConnection(); + +// Close all connections +await driver.destroy(); +``` + +## Performance Optimization + +### Indexes + +```typescript +// Create index migration +export async function up(knex) { + await knex.schema.table('objectstack_opportunity', (table) => { + table.index('account_id'); + table.index('stage'); + table.index(['created_at', 'stage']); // Composite index + }); +} +``` + +### Query Optimization + +```typescript +// Use explain to analyze queries +const plan = await driver.raw('EXPLAIN ANALYZE SELECT ...'); + +// Create covering indexes for frequently accessed columns +// Use partial indexes for filtered queries (PostgreSQL) +await knex.raw(` + CREATE INDEX idx_active_opportunities + ON objectstack_opportunity(account_id, amount) + WHERE stage NOT IN ('closed_won', 'closed_lost') +`); +``` + +## Best Practices + +1. **Connection Pooling**: Configure appropriate pool size based on load +2. **Migrations**: Always use migrations for schema changes, never raw DDL +3. **Transactions**: Use transactions for multi-step operations +4. **Prepared Statements**: Use parameterized queries to prevent SQL injection +5. **Indexes**: Create indexes on frequently queried fields +6. **Monitoring**: Monitor slow query logs and connection pool metrics +7. **Backups**: Implement regular database backups + +## Environment-Specific Configuration + +```typescript +// config/database.ts +export const getDatabaseConfig = () => { + const env = process.env.NODE_ENV || 'development'; + + const configs = { + development: { + client: 'better-sqlite3', + connection: { filename: './data/dev.db' }, + useNullAsDefault: true, + debug: true, + }, + test: { + client: 'better-sqlite3', + connection: { filename: ':memory:' }, + useNullAsDefault: true, + }, + production: { + client: 'pg', + connection: process.env.DATABASE_URL, + pool: { min: 2, max: 10 }, + ssl: { rejectUnauthorized: false }, + }, + }; + + return configs[env] || configs.development; +}; + +const stack = defineStack({ + driver: DriverSQL.configure(getDatabaseConfig()), +}); +``` + +## Troubleshooting + +### Connection Issues + +```typescript +// Test database connection +try { + await driver.checkConnection(); + console.log('Database connected successfully'); +} catch (error) { + console.error('Database connection failed:', error); +} +``` + +### Migration Errors + +```bash +# Check migration status +npx knex migrate:status + +# Rollback and re-run +npx knex migrate:rollback +npx knex migrate:latest +``` + +### Query Debugging + +```typescript +// Enable query logging +const stack = defineStack({ + driver: DriverSQL.configure({ + client: 'pg', + connection: { /* ... */ }, + debug: true, // Log all queries + }), +}); +``` + +## Deployment + +### Heroku PostgreSQL + +```bash +# Heroku automatically provides DATABASE_URL +heroku addons:create heroku-postgresql:hobby-dev + +# Run migrations on deployment +echo "npx knex migrate:latest" > Procfile.release +``` + +### Railway PostgreSQL + +```bash +# Use Railway's DATABASE_URL +railway up +``` + +### Vercel PostgreSQL + +```typescript +// Vercel uses connection pooling +import { createClient } from '@vercel/postgres'; + +const stack = defineStack({ + driver: DriverSQL.configure({ + client: 'pg', + connection: process.env.POSTGRES_URL, + }), +}); +``` + +## License + +Apache-2.0 + +## See Also + +- [Knex.js Documentation](https://knexjs.org/) +- [PostgreSQL Documentation](https://www.postgresql.org/docs/) +- [MySQL Documentation](https://dev.mysql.com/doc/) +- [@objectstack/driver-turso](../driver-turso/) - Edge-first SQLite alternative +- [@objectstack/driver-memory](../driver-memory/) - In-memory driver for testing diff --git a/packages/plugins/plugin-audit/README.md b/packages/plugins/plugin-audit/README.md new file mode 100644 index 000000000..81720f51d --- /dev/null +++ b/packages/plugins/plugin-audit/README.md @@ -0,0 +1,414 @@ +# @objectstack/plugin-audit + +Audit Plugin for ObjectStack — System audit log object and audit trail for compliance and security monitoring. + +## Features + +- **Comprehensive Audit Trail**: Track all CRUD operations across all objects +- **User Activity Logging**: Record who did what, when, and from where +- **Field-Level Changes**: Capture before/after values for all field changes +- **Compliance Ready**: Meet SOC 2, HIPAA, GDPR audit requirements +- **Query Filtering**: Search audit logs by user, object, action, date range +- **Retention Policies**: Auto-archive or delete old audit logs +- **Security Events**: Track authentication, authorization, and security-related events +- **Immutable Logs**: Audit records cannot be modified or deleted (only archived) + +## Installation + +```bash +pnpm add @objectstack/plugin-audit +``` + +## Basic Usage + +```typescript +import { defineStack } from '@objectstack/spec'; +import { PluginAudit } from '@objectstack/plugin-audit'; + +const stack = defineStack({ + plugins: [ + PluginAudit.configure({ + enabled: true, + trackObjects: '*', // Track all objects + trackFields: '*', // Track all field changes + }), + ], +}); +``` + +## Configuration + +```typescript +interface AuditPluginConfig { + /** Enable audit logging (default: true) */ + enabled?: boolean; + + /** Objects to track ('*' for all, or array of object names) */ + trackObjects?: '*' | string[]; + + /** Fields to track ('*' for all, or object-specific config) */ + trackFields?: '*' | Record; + + /** Track system events (login, logout, failed auth, etc.) */ + trackSystemEvents?: boolean; + + /** Retention period in days (default: 365) */ + retentionDays?: number; + + /** Auto-archive after retention period (default: true) */ + autoArchive?: boolean; + + /** Exclude certain users from audit (e.g., system users) */ + excludeUsers?: string[]; +} +``` + +## Audit Log Schema + +The plugin automatically creates the `audit_log` object: + +```typescript +{ + id: string; // Unique audit log entry ID + timestamp: datetime; // When the action occurred + userId: string; // User who performed the action + userName: string; // User's name at time of action + userEmail: string; // User's email at time of action + action: string; // 'insert', 'update', 'delete', 'read' + object: string; // Object type (e.g., 'opportunity') + recordId: string; // Record ID that was affected + recordName: string; // Record display name + changes: json; // Field-level changes (before/after) + metadata: json; // Additional context (IP, user agent, etc.) + ipAddress: string; // Client IP address + userAgent: string; // Client user agent + sessionId: string; // Session ID + status: string; // 'success' | 'failed' + errorMessage: string; // Error message if action failed +} +``` + +## Automatic Audit Logging + +All CRUD operations are automatically audited: + +```typescript +// This operation is automatically audited +await kernel.getDriver().insert({ + object: 'opportunity', + data: { + name: 'Big Deal', + amount: 100000, + stage: 'prospecting', + }, +}); + +// Audit log entry created: +// { +// action: 'insert', +// object: 'opportunity', +// recordId: '123', +// recordName: 'Big Deal', +// changes: { +// name: { from: null, to: 'Big Deal' }, +// amount: { from: null, to: 100000 }, +// stage: { from: null, to: 'prospecting' } +// } +// } +``` + +## Querying Audit Logs + +```typescript +// Get audit logs via kernel +const auditService = kernel.getService('audit'); + +// Get all changes for a specific record +const recordHistory = await auditService.getRecordHistory({ + object: 'opportunity', + recordId: '123', +}); + +// Get user activity +const userActivity = await auditService.getUserActivity({ + userId: 'user:456', + from: '2024-01-01', + to: '2024-01-31', +}); + +// Search audit logs +const logs = await auditService.searchLogs({ + action: 'delete', + object: 'account', + from: '2024-01-01', + to: '2024-01-31', +}); + +// Get failed actions (security monitoring) +const failures = await auditService.getFailedActions({ + from: '2024-01-01', + limit: 100, +}); +``` + +## Selective Tracking + +### Track Specific Objects + +```typescript +PluginAudit.configure({ + trackObjects: ['opportunity', 'account', 'contact'], + // Only these objects will be audited +}); +``` + +### Track Specific Fields + +```typescript +PluginAudit.configure({ + trackFields: { + opportunity: ['stage', 'amount', 'close_date'], // Only track these fields + account: '*', // Track all fields for accounts + contact: ['email', 'phone'], // Only email and phone for contacts + }, +}); +``` + +### Exclude System Users + +```typescript +PluginAudit.configure({ + excludeUsers: [ + 'system:integration', + 'system:cron', + 'service:automation', + ], + // These users' actions won't be audited +}); +``` + +## Advanced Features + +### Manual Audit Entries + +```typescript +// Log custom security event +await auditService.log({ + action: 'security:password_reset', + userId: 'user:456', + metadata: { + resetMethod: 'email', + ipAddress: '192.168.1.100', + }, +}); + +// Log business process event +await auditService.log({ + action: 'workflow:approval', + object: 'opportunity', + recordId: '123', + metadata: { + approver: 'manager:789', + decision: 'approved', + }, +}); +``` + +### Audit Snapshots + +```typescript +// Create snapshot of record state at a specific time +const snapshot = await auditService.getRecordSnapshot({ + object: 'opportunity', + recordId: '123', + asOf: '2024-01-15T10:30:00Z', +}); + +// Returns: Record state as it existed on Jan 15, 2024 at 10:30 AM +``` + +### Audit Reports + +```typescript +// Generate compliance report +const report = await auditService.generateReport({ + type: 'compliance', + from: '2024-01-01', + to: '2024-03-31', + format: 'pdf', +}); + +// Generate user activity report +const userReport = await auditService.generateReport({ + type: 'user_activity', + userId: 'user:456', + from: '2024-01-01', + to: '2024-01-31', + includeDetails: true, +}); +``` + +### Data Retention & Archival + +```typescript +// Archive old audit logs +await auditService.archiveLogs({ + olderThan: '2023-01-01', + destination: 's3://audit-archive/2023/', +}); + +// Permanently delete archived logs (compliance approved) +await auditService.purgeLogs({ + olderThan: '2020-01-01', + confirmed: true, // Safety check +}); +``` + +## Security Events + +The plugin automatically logs security-related events: + +- **Authentication**: Login, logout, password changes +- **Authorization**: Permission denied, role changes +- **Data Access**: Read operations on sensitive fields +- **Configuration**: System setting changes +- **API**: API key usage, rate limit violations + +```typescript +// Automatically logged: +// - Login attempts (success/failure) +// - Permission denied errors +// - Sensitive field access +// - API authentication failures +``` + +## Compliance Integration + +### GDPR Compliance + +```typescript +// Right to be forgotten - audit trail for deletions +await auditService.logDataDeletion({ + userId: 'user:456', + reason: 'gdpr_right_to_be_forgotten', + deletedRecords: [ + { object: 'user', recordId: '456' }, + { object: 'contact', recordId: '789' }, + ], +}); + +// Right of access - audit trail for data exports +await auditService.logDataExport({ + userId: 'user:456', + reason: 'gdpr_data_access_request', + exportedObjects: ['user', 'contact', 'activity'], +}); +``` + +### SOC 2 Compliance + +```typescript +// Log administrative actions +await auditService.logAdminAction({ + adminId: 'admin:123', + action: 'user_role_change', + targetUserId: 'user:456', + changes: { + roles: { from: ['user'], to: ['user', 'admin'] }, + }, +}); +``` + +### HIPAA Compliance + +```typescript +// Track PHI access +PluginAudit.configure({ + trackFields: { + patient: ['ssn', 'medical_record_number', 'diagnosis'], // PHI fields + }, + trackSystemEvents: true, +}); +``` + +## REST API Endpoints + +``` +GET /api/v1/audit # List audit logs +GET /api/v1/audit/:id # Get specific log entry +GET /api/v1/audit/record/:object/:id # Get record history +GET /api/v1/audit/user/:userId # Get user activity +POST /api/v1/audit/report # Generate report +POST /api/v1/audit/archive # Archive old logs +``` + +## Dashboard Integration + +```typescript +// Audit dashboard widget +const auditWidget = { + title: 'Recent Activity', + query: { + object: 'audit_log', + limit: 50, + sort: [{ field: 'timestamp', direction: 'desc' }], + }, +}; + +// Security alerts dashboard +const securityWidget = { + title: 'Failed Login Attempts', + query: { + object: 'audit_log', + filters: [ + { field: 'action', operator: 'eq', value: 'auth:login' }, + { field: 'status', operator: 'eq', value: 'failed' }, + { field: 'timestamp', operator: 'gte', value: 'today' }, + ], + }, +}; +``` + +## Best Practices + +1. **Selective Tracking**: Only audit what's necessary for compliance +2. **Retention Policy**: Set appropriate retention based on regulations +3. **Performance**: Archive old logs to keep query performance high +4. **Security**: Restrict access to audit logs (admin only) +5. **Immutability**: Never allow modification of audit records +6. **Monitoring**: Set alerts for suspicious activity patterns +7. **Regular Review**: Periodically review audit logs for anomalies + +## Performance Considerations + +- **Async Logging**: Audit writes happen asynchronously (no performance impact) +- **Indexing**: Automatically indexes `userId`, `object`, `recordId`, `timestamp` +- **Partitioning**: Consider table partitioning for high-volume deployments +- **Archival**: Move old logs to cold storage (S3, Glacier) + +## Contract Implementation + +Implements audit logging hooks from `@objectstack/spec/contracts`: + +```typescript +interface IAuditService { + log(entry: AuditLogEntry): Promise; + getRecordHistory(options: RecordHistoryOptions): Promise; + getUserActivity(options: UserActivityOptions): Promise; + searchLogs(filter: AuditLogFilter): Promise; + getRecordSnapshot(options: SnapshotOptions): Promise; + archiveLogs(options: ArchiveOptions): Promise; +} +``` + +## License + +Apache-2.0 + +## See Also + +- [SOC 2 Compliance Guide](https://www.aicpa.org/interestareas/frc/assuranceadvisoryservices/sorhome) +- [GDPR Requirements](https://gdpr.eu/) +- [HIPAA Compliance](https://www.hhs.gov/hipaa/) +- [@objectstack/plugin-security](../plugin-security/) +- [Audit Logging Best Practices](/content/docs/guides/audit/) diff --git a/packages/plugins/plugin-mcp-server/README.md b/packages/plugins/plugin-mcp-server/README.md new file mode 100644 index 000000000..e60795838 --- /dev/null +++ b/packages/plugins/plugin-mcp-server/README.md @@ -0,0 +1,528 @@ +# @objectstack/plugin-mcp-server + +MCP Runtime Server Plugin for ObjectStack — exposes AI tools, data resources, and agent prompts via the Model Context Protocol. + +## Features + +- **Model Context Protocol (MCP)**: Expose ObjectStack resources to AI models via MCP +- **AI Tools**: Auto-generate MCP tools from ObjectStack actions and flows +- **Data Resources**: Expose objects, records, and metadata as MCP resources +- **Agent Prompts**: Register prompt templates for AI agents +- **Type-Safe**: Full Zod schema validation for tool inputs/outputs +- **Auto-Discovery**: MCP clients automatically discover available tools and resources +- **Streaming Support**: Stream large datasets and real-time updates +- **Security**: Built-in permission checks for tool execution + +## What is MCP? + +Model Context Protocol (MCP) is an open protocol that standardizes how AI applications provide context to Large Language Models (LLMs). It allows AI models to: + +- **Access Tools**: Execute functions and operations +- **Read Resources**: Access data and content +- **Use Prompts**: Leverage pre-defined prompt templates + +Read more: [MCP Specification](https://modelcontextprotocol.io/) + +## Installation + +```bash +pnpm add @objectstack/plugin-mcp-server +``` + +## Basic Usage + +```typescript +import { defineStack } from '@objectstack/spec'; +import { PluginMCPServer } from '@objectstack/plugin-mcp-server'; + +const stack = defineStack({ + plugins: [ + PluginMCPServer.configure({ + serverName: 'objectstack-server', + version: '1.0.0', + autoRegisterTools: true, + }), + ], +}); +``` + +## Configuration + +```typescript +interface MCPServerConfig { + /** Server name (shown to AI clients) */ + serverName?: string; + + /** Server version */ + version?: string; + + /** Auto-register tools from actions and flows */ + autoRegisterTools?: boolean; + + /** Auto-expose objects as resources */ + autoExposeObjects?: boolean; + + /** Enable streaming for large responses */ + enableStreaming?: boolean; + + /** Transport mechanism ('stdio' | 'http') */ + transport?: 'stdio' | 'http'; + + /** HTTP port (if transport is 'http') */ + port?: number; +} +``` + +## MCP Tools + +### Auto-Generated Tools + +ObjectStack automatically exposes these operations as MCP tools: + +```typescript +// CRUD operations (auto-registered) +'objectstack_find' // Query records +'objectstack_findOne' // Get single record +'objectstack_create' // Create record +'objectstack_update' // Update record +'objectstack_delete' // Delete record + +// Metadata operations +'objectstack_describeObject' // Get object schema +'objectstack_listObjects' // List all objects +'objectstack_listFields' // List object fields +``` + +### Custom Tools + +Register custom tools that AI models can call: + +```typescript +import { defineTool } from '@objectstack/spec'; + +const calculateRevenueTool = defineTool({ + name: 'calculate_revenue', + description: 'Calculate total revenue for an account', + inputSchema: { + type: 'object', + properties: { + accountId: { type: 'string', description: 'Account ID' }, + startDate: { type: 'string', description: 'Start date (ISO 8601)' }, + endDate: { type: 'string', description: 'End date (ISO 8601)' }, + }, + required: ['accountId'], + }, + async execute({ accountId, startDate, endDate }) { + const opportunities = await kernel.getDriver().find({ + object: 'opportunity', + filters: [ + { field: 'account_id', operator: 'eq', value: accountId }, + { field: 'stage', operator: 'eq', value: 'closed_won' }, + { field: 'close_date', operator: 'gte', value: startDate }, + { field: 'close_date', operator: 'lte', value: endDate }, + ], + }); + + const total = opportunities.reduce((sum, opp) => sum + opp.amount, 0); + + return { + accountId, + totalRevenue: total, + opportunityCount: opportunities.length, + }; + }, +}); + +// Register with MCP server +kernel.getService('mcp').registerTool(calculateRevenueTool); +``` + +## MCP Resources + +### Auto-Exposed Objects + +All ObjectStack objects are automatically exposed as MCP resources: + +``` +objectstack://objects/opportunity # Opportunity object schema +objectstack://objects/opportunity/records # All opportunity records +objectstack://objects/opportunity/123 # Specific opportunity record +``` + +### Custom Resources + +Expose custom resources to AI models: + +```typescript +kernel.getService('mcp').registerResource({ + uri: 'objectstack://reports/sales-pipeline', + name: 'Sales Pipeline Report', + description: 'Current sales pipeline with stages and amounts', + mimeType: 'application/json', + async read() { + const opportunities = await kernel.getDriver().find({ + object: 'opportunity', + filters: [ + { field: 'stage', operator: 'neq', value: 'closed_won' }, + { field: 'stage', operator: 'neq', value: 'closed_lost' }, + ], + }); + + const pipeline = opportunities.reduce((acc, opp) => { + acc[opp.stage] = (acc[opp.stage] || 0) + opp.amount; + return acc; + }, {}); + + return { + content: [ + { + type: 'text', + text: JSON.stringify(pipeline, null, 2), + }, + ], + }; + }, +}); +``` + +## MCP Prompts + +Register prompt templates that AI models can use: + +```typescript +kernel.getService('mcp').registerPrompt({ + name: 'analyze_account', + description: 'Analyze an account and its opportunities', + arguments: [ + { + name: 'accountId', + description: 'Account ID to analyze', + required: true, + }, + ], + async render({ accountId }) { + const account = await kernel.getDriver().findOne({ + object: 'account', + filters: [{ field: 'id', operator: 'eq', value: accountId }], + }); + + const opportunities = await kernel.getDriver().find({ + object: 'opportunity', + filters: [{ field: 'account_id', operator: 'eq', value: accountId }], + }); + + return { + messages: [ + { + role: 'user', + content: { + type: 'text', + text: `Analyze this account and provide insights: + +Account: ${account.name} +Industry: ${account.industry} +Total Opportunities: ${opportunities.length} +Total Value: $${opportunities.reduce((sum, o) => sum + o.amount, 0)} + +Opportunities: +${opportunities.map(o => `- ${o.name} (${o.stage}): $${o.amount}`).join('\n')} + +Please provide: +1. Key insights about this account +2. Risk assessment +3. Recommendations for next steps`, + }, + }, + ], + }; + }, +}); +``` + +## Using with AI Clients + +### Claude Desktop + +Add to `~/Library/Application Support/Claude/claude_desktop_config.json`: + +```json +{ + "mcpServers": { + "objectstack": { + "command": "node", + "args": ["/path/to/your/objectstack/server.js"], + "env": { + "DATABASE_URL": "your-database-url" + } + } + } +} +``` + +### Cursor IDE + +Add to `.cursor/mcp.json`: + +```json +{ + "mcpServers": { + "objectstack": { + "command": "node", + "args": ["./server.js"] + } + } +} +``` + +### Cline VS Code Extension + +Configure in Cline settings: + +```json +{ + "cline.mcpServers": { + "objectstack": { + "command": "node", + "args": ["./server.js"] + } + } +} +``` + +## Server Implementation + +### Stdio Transport (Default) + +```typescript +// server.ts +import { defineStack } from '@objectstack/spec'; +import { PluginMCPServer } from '@objectstack/plugin-mcp-server'; +import { DriverTurso } from '@objectstack/driver-turso'; + +const stack = defineStack({ + driver: DriverTurso.configure({ + url: process.env.DATABASE_URL!, + authToken: process.env.TURSO_AUTH_TOKEN!, + }), + plugins: [ + PluginMCPServer.configure({ + serverName: 'my-crm', + transport: 'stdio', // Claude Desktop, Cursor, Cline + }), + ], +}); + +await stack.boot(); +``` + +### HTTP Transport + +```typescript +const stack = defineStack({ + driver: DriverTurso.configure({ /* ... */ }), + plugins: [ + PluginMCPServer.configure({ + serverName: 'my-crm', + transport: 'http', + port: 3100, + }), + ], +}); + +await stack.boot(); +// MCP server running on http://localhost:3100 +``` + +## Advanced Features + +### Streaming Resources + +```typescript +kernel.getService('mcp').registerResource({ + uri: 'objectstack://exports/opportunities-csv', + name: 'Opportunities Export (CSV)', + mimeType: 'text/csv', + async *stream() { + // Stream header + yield 'Name,Stage,Amount,Close Date\n'; + + // Stream records in batches + let offset = 0; + const batchSize = 100; + + while (true) { + const batch = await kernel.getDriver().find({ + object: 'opportunity', + limit: batchSize, + offset, + }); + + if (batch.length === 0) break; + + for (const opp of batch) { + yield `${opp.name},${opp.stage},${opp.amount},${opp.close_date}\n`; + } + + offset += batchSize; + } + }, +}); +``` + +### Tool Permissions + +```typescript +kernel.getService('mcp').registerTool({ + name: 'delete_opportunity', + description: 'Delete an opportunity', + permissions: ['opportunity:delete'], // Require permission + inputSchema: { + type: 'object', + properties: { + id: { type: 'string' }, + }, + required: ['id'], + }, + async execute({ id }, context) { + // context includes userId, permissions, etc. + if (!context.hasPermission('opportunity:delete')) { + throw new Error('Permission denied'); + } + + await kernel.getDriver().delete({ + object: 'opportunity', + filters: [{ field: 'id', operator: 'eq', value: id }], + }); + + return { success: true, deleted: id }; + }, +}); +``` + +### Dynamic Tool Registration + +```typescript +// Register tools from flow definitions +const flows = await kernel.getMetadata('flow'); + +for (const flow of flows) { + kernel.getService('mcp').registerTool({ + name: `flow_${flow.name}`, + description: flow.description, + inputSchema: generateSchemaFromFlow(flow), + async execute(inputs) { + return await kernel.executeFlow(flow.name, inputs); + }, + }); +} +``` + +## Server Capabilities + +The MCP server exposes these capabilities: + +```json +{ + "capabilities": { + "tools": { + "listChanged": true + }, + "resources": { + "subscribe": true, + "listChanged": true + }, + "prompts": { + "listChanged": true + }, + "logging": {}, + "experimental": { + "streaming": true + } + } +} +``` + +## Best Practices + +1. **Tool Design**: Keep tools focused and well-documented +2. **Resource Naming**: Use clear, hierarchical URI schemes +3. **Prompt Templates**: Make prompts flexible with arguments +4. **Error Handling**: Always return helpful error messages +5. **Permissions**: Check permissions before tool execution +6. **Performance**: Use streaming for large datasets +7. **Versioning**: Version your server and tools + +## Debugging + +Enable debug logging: + +```typescript +PluginMCPServer.configure({ + serverName: 'my-crm', + debug: true, // Log all MCP messages +}); +``` + +View MCP messages in client: +- **Claude Desktop**: Check logs in `~/Library/Logs/Claude/mcp*.log` +- **Cursor**: Check Output panel → MCP Server +- **Cline**: Check extension logs + +## Example: Complete CRM Server + +```typescript +import { defineStack, defineTool } from '@objectstack/spec'; +import { PluginMCPServer } from '@objectstack/plugin-mcp-server'; + +const stack = defineStack({ + driver: /* ... */, + plugins: [ + PluginMCPServer.configure({ + serverName: 'crm-assistant', + autoRegisterTools: true, + }), + ], +}); + +await stack.boot(); + +const mcp = stack.kernel.getService('mcp'); + +// Register custom tools +mcp.registerTool(defineTool({ + name: 'forecast_revenue', + description: 'Forecast revenue based on pipeline', + async execute() { + // Implementation + }, +})); + +// Register custom resources +mcp.registerResource({ + uri: 'objectstack://dashboards/sales', + name: 'Sales Dashboard', + async read() { + // Implementation + }, +}); + +// Register prompts +mcp.registerPrompt({ + name: 'weekly_report', + description: 'Generate weekly sales report', + async render() { + // Implementation + }, +}); +``` + +## License + +Apache-2.0 + +## See Also + +- [Model Context Protocol Specification](https://modelcontextprotocol.io/) +- [MCP TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk) +- [@objectstack/spec/ai](../../spec/src/ai/) +- [Building MCP Servers Guide](/content/docs/guides/mcp/) diff --git a/packages/services/service-ai/README.md b/packages/services/service-ai/README.md new file mode 100644 index 000000000..76dd2b7cc --- /dev/null +++ b/packages/services/service-ai/README.md @@ -0,0 +1,293 @@ +# @objectstack/service-ai + +AI Service for ObjectStack — implements `IAIService` with LLM adapter layer, conversation management, tool registry, and REST/SSE routes. + +## Features + +- **Multi-Provider LLM Support**: Supports OpenAI, Anthropic, Google, and custom gateway providers via Vercel AI SDK +- **Conversation Management**: Track and manage AI conversations with full history +- **Tool Registry**: Register and execute tools that AI agents can call +- **Streaming Support**: Real-time streaming responses via Server-Sent Events (SSE) +- **REST API**: Auto-generated endpoints for AI operations +- **Type-Safe**: Full TypeScript support with type inference + +## Installation + +```bash +pnpm add @objectstack/service-ai +``` + +### Peer Dependencies + +Install the LLM provider(s) you need: + +```bash +# OpenAI +pnpm add @ai-sdk/openai + +# Anthropic (Claude) +pnpm add @ai-sdk/anthropic + +# Google (Gemini) +pnpm add @ai-sdk/google + +# Custom Gateway +pnpm add @ai-sdk/gateway +``` + +All peer dependencies are optional — install only what you need. + +## Basic Usage + +```typescript +import { defineStack } from '@objectstack/spec'; +import { ServiceAI } from '@objectstack/service-ai'; +import { openai } from '@ai-sdk/openai'; + +const stack = defineStack({ + services: [ + ServiceAI.configure({ + models: { + default: openai('gpt-4'), + fast: openai('gpt-3.5-turbo'), + }, + }), + ], +}); +``` + +## Configuration + +```typescript +interface AIServiceConfig { + /** Map of model IDs to AI SDK model instances */ + models: Record; + + /** Default model to use when not specified */ + defaultModel?: string; + + /** Maximum conversation history length */ + maxHistoryLength?: number; + + /** Enable streaming responses (default: true) */ + enableStreaming?: boolean; +} +``` + +## Service API + +The `IAIService` interface provides: + +### Conversation Management + +```typescript +// Get AI service from kernel +const ai = kernel.getService('ai'); + +// Create a new conversation +const conversation = await ai.createConversation({ + model: 'default', + systemPrompt: 'You are a helpful assistant.', +}); + +// Send a message +const response = await ai.sendMessage({ + conversationId: conversation.id, + message: 'What is ObjectStack?', +}); + +// Stream a message +const stream = await ai.streamMessage({ + conversationId: conversation.id, + message: 'Explain in detail...', +}); + +for await (const chunk of stream) { + process.stdout.write(chunk.text); +} +``` + +### Tool Registry + +```typescript +// Register a tool +ai.registerTool({ + name: 'get_weather', + description: 'Get current weather for a location', + parameters: { + type: 'object', + properties: { + location: { type: 'string', description: 'City name' }, + }, + required: ['location'], + }, + execute: async ({ location }) => { + // Your tool implementation + return { temperature: 72, condition: 'sunny' }; + }, +}); + +// Tools are automatically available to AI agents +const response = await ai.sendMessage({ + conversationId: conversation.id, + message: 'What is the weather in San Francisco?', +}); +``` + +## REST API Endpoints + +When used with `@objectstack/rest`, the following endpoints are auto-generated: + +``` +POST /api/v1/ai/conversations # Create conversation +GET /api/v1/ai/conversations/:id # Get conversation +POST /api/v1/ai/conversations/:id/messages # Send message +GET /api/v1/ai/conversations/:id/stream # Stream response (SSE) +GET /api/v1/ai/tools # List available tools +POST /api/v1/ai/tools/:name/execute # Execute a tool +``` + +## Multi-Model Configuration + +```typescript +import { openai } from '@ai-sdk/openai'; +import { anthropic } from '@ai-sdk/anthropic'; +import { google } from '@ai-sdk/google'; + +const stack = defineStack({ + services: [ + ServiceAI.configure({ + models: { + // Fast model for simple tasks + fast: openai('gpt-3.5-turbo'), + + // Default model for general use + default: openai('gpt-4'), + + // Advanced reasoning + reasoning: openai('gpt-4-turbo'), + + // Anthropic Claude + claude: anthropic('claude-3-opus-20240229'), + + // Google Gemini + gemini: google('gemini-pro'), + }, + defaultModel: 'default', + }), + ], +}); +``` + +## Advanced Features + +### Custom System Prompts + +```typescript +const conversation = await ai.createConversation({ + model: 'default', + systemPrompt: `You are an expert in ObjectStack. + Answer questions about the framework accurately and concisely. + Always provide code examples when relevant.`, +}); +``` + +### Conversation History + +```typescript +// Get full conversation history +const history = await ai.getConversationHistory(conversationId); + +// Clear conversation history +await ai.clearConversation(conversationId); + +// Delete conversation +await ai.deleteConversation(conversationId); +``` + +### Error Handling + +```typescript +try { + const response = await ai.sendMessage({ + conversationId: conversation.id, + message: 'Hello', + }); +} catch (error) { + if (error.code === 'RATE_LIMIT_EXCEEDED') { + // Handle rate limiting + } else if (error.code === 'MODEL_NOT_FOUND') { + // Handle missing model + } +} +``` + +## Integration with ObjectStack Agents + +The AI service integrates with the `@objectstack/spec/ai` agent protocol: + +```typescript +import { defineAgent } from '@objectstack/spec'; + +const myAgent = defineAgent({ + name: 'code_assistant', + model: 'default', + systemPrompt: 'You help write ObjectStack code.', + tools: ['create_object', 'create_field', 'create_view'], +}); + +// Agent automatically uses the AI service +const result = await myAgent.execute({ + input: 'Create a contact object with name and email fields', +}); +``` + +## Architecture + +The service follows a layered architecture: + +1. **Adapter Layer**: Abstracts different LLM providers (OpenAI, Anthropic, Google) +2. **Conversation Manager**: Handles conversation state and history +3. **Tool Registry**: Manages tool registration and execution +4. **REST Routes**: Auto-generated HTTP endpoints +5. **SSE Streaming**: Real-time streaming support + +## Contract Implementation + +Implements `IAIService` from `@objectstack/spec/contracts`: + +```typescript +interface IAIService { + createConversation(options: ConversationOptions): Promise; + sendMessage(options: MessageOptions): Promise; + streamMessage(options: MessageOptions): AsyncIterable; + registerTool(tool: AITool): void; + getConversationHistory(conversationId: string): Promise; + deleteConversation(conversationId: string): Promise; +} +``` + +## Performance Considerations + +- **Streaming**: Use streaming for long responses to improve perceived performance +- **Model Selection**: Choose appropriate models based on task complexity +- **Conversation History**: Limit history length to control token usage +- **Caching**: Provider-level caching is handled automatically + +## Best Practices + +1. **Model Selection**: Use fast models for simple tasks, advanced models for complex reasoning +2. **System Prompts**: Provide clear, specific instructions in system prompts +3. **Tool Design**: Keep tools focused and well-documented +4. **Error Handling**: Always handle rate limits and API errors gracefully +5. **Streaming**: Use streaming for better UX on long-running queries + +## License + +Apache-2.0 + +## See Also + +- [Vercel AI SDK Documentation](https://sdk.vercel.ai/docs) +- [@objectstack/spec/ai Protocol](../../spec/src/ai/) +- [AI Agent Guide](/content/docs/guides/ai/) diff --git a/packages/services/service-analytics/README.md b/packages/services/service-analytics/README.md new file mode 100644 index 000000000..eb3e05fea --- /dev/null +++ b/packages/services/service-analytics/README.md @@ -0,0 +1,395 @@ +# @objectstack/service-analytics + +Analytics Service for ObjectStack — implements `IAnalyticsService` with multi-driver strategy pattern (NativeSQL, ObjectQL, InMemory). + +## Features + +- **Multi-Driver Architecture**: Choose the right execution strategy for your analytics queries + - **NativeSQL**: Direct SQL execution for maximum performance on large datasets + - **ObjectQL**: Leverage ObjectStack's query engine for metadata-aware analytics + - **InMemory**: Fast aggregations on small datasets without database round-trips +- **Aggregation Functions**: SUM, COUNT, AVG, MIN, MAX, GROUP BY, HAVING +- **Time Series Analysis**: Time-based aggregations and grouping +- **Custom Metrics**: Define and track custom business metrics +- **Dashboard Integration**: Auto-generated REST endpoints for visualization +- **Type-Safe**: Full TypeScript support with inferred result types + +## Installation + +```bash +pnpm add @objectstack/service-analytics +``` + +## Basic Usage + +```typescript +import { defineStack } from '@objectstack/spec'; +import { ServiceAnalytics } from '@objectstack/service-analytics'; + +const stack = defineStack({ + services: [ + ServiceAnalytics.configure({ + defaultDriver: 'objectql', // or 'sql', 'memory' + enableCaching: true, + }), + ], +}); +``` + +## Configuration + +```typescript +interface AnalyticsServiceConfig { + /** Default execution driver */ + defaultDriver?: 'sql' | 'objectql' | 'memory'; + + /** Enable query result caching */ + enableCaching?: boolean; + + /** Cache TTL in seconds (default: 300) */ + cacheTTL?: number; + + /** Maximum result set size for in-memory driver */ + maxMemoryResults?: number; +} +``` + +## Service API + +```typescript +// Get analytics service from kernel +const analytics = kernel.getService('analytics'); +``` + +### Basic Aggregations + +```typescript +// Count records +const totalOrders = await analytics.count({ + object: 'order', + filters: [{ field: 'status', operator: 'eq', value: 'completed' }], +}); + +// Sum field values +const totalRevenue = await analytics.sum({ + object: 'order', + field: 'amount', + filters: [{ field: 'created_at', operator: 'gte', value: '2024-01-01' }], +}); + +// Calculate average +const avgOrderValue = await analytics.avg({ + object: 'order', + field: 'amount', +}); + +// Find min/max +const highestOrder = await analytics.max({ + object: 'order', + field: 'amount', +}); +``` + +### Group By Aggregations + +```typescript +// Revenue by product category +const revenueByCategory = await analytics.groupBy({ + object: 'order_item', + groupBy: ['product.category'], + aggregations: [ + { function: 'sum', field: 'total', as: 'revenue' }, + { function: 'count', as: 'order_count' }, + ], +}); + +// Result format: +// [ +// { category: 'Electronics', revenue: 125000, order_count: 342 }, +// { category: 'Clothing', revenue: 98000, order_count: 567 }, +// ] +``` + +### Time Series Analytics + +```typescript +// Daily revenue for the past 30 days +const dailyRevenue = await analytics.timeSeries({ + object: 'order', + dateField: 'created_at', + interval: 'day', + aggregations: [ + { function: 'sum', field: 'amount', as: 'revenue' }, + { function: 'count', as: 'orders' }, + ], + filters: [ + { + field: 'created_at', + operator: 'gte', + value: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000), + }, + ], +}); + +// Result format: +// [ +// { date: '2024-01-01', revenue: 12500, orders: 45 }, +// { date: '2024-01-02', revenue: 15200, orders: 52 }, +// ] +``` + +### Custom Metrics + +```typescript +// Define a metric +analytics.defineMetric({ + name: 'monthly_recurring_revenue', + description: 'MRR from active subscriptions', + calculation: { + object: 'subscription', + aggregation: 'sum', + field: 'amount', + filters: [{ field: 'status', operator: 'eq', value: 'active' }], + }, +}); + +// Query the metric +const mrr = await analytics.getMetric('monthly_recurring_revenue'); +``` + +## Multi-Driver Strategy + +### When to Use Each Driver + +#### NativeSQL Driver +**Best for**: Large datasets, complex joins, database-specific optimizations + +```typescript +const result = await analytics.query({ + driver: 'sql', + object: 'order', + aggregations: [{ function: 'sum', field: 'amount' }], + groupBy: ['customer_id'], + having: [{ field: 'sum_amount', operator: 'gt', value: 10000 }], +}); +``` + +**Advantages:** +- Direct SQL execution for maximum performance +- Leverages database indexes and query optimization +- Handles millions of records efficiently + +**Limitations:** +- Bypasses ObjectStack metadata layer +- May miss field-level transformations +- Less portable across databases + +#### ObjectQL Driver +**Best for**: Metadata-aware analytics, cross-object aggregations + +```typescript +const result = await analytics.query({ + driver: 'objectql', + object: 'opportunity', + aggregations: [ + { function: 'sum', field: 'amount' }, + { function: 'count' }, + ], + groupBy: ['account.industry'], +}); +``` + +**Advantages:** +- Respects object/field metadata and permissions +- Handles formula fields and computed values +- Consistent with ObjectQL query behavior + +**Limitations:** +- Slightly slower than direct SQL +- Additional abstraction layer + +#### InMemory Driver +**Best for**: Small datasets, pre-filtered results, real-time dashboards + +```typescript +const result = await analytics.query({ + driver: 'memory', + object: 'task', + aggregations: [{ function: 'count' }], + groupBy: ['status'], +}); +``` + +**Advantages:** +- Zero database round-trips for cached data +- Instant results for small datasets +- Useful for client-side analytics + +**Limitations:** +- Limited to `maxMemoryResults` (default: 10,000) +- Requires data to be loaded into memory first + +## REST API Endpoints + +When used with `@objectstack/rest`: + +``` +POST /api/v1/analytics/count # Count records +POST /api/v1/analytics/sum # Sum field values +POST /api/v1/analytics/avg # Calculate average +POST /api/v1/analytics/min # Find minimum +POST /api/v1/analytics/max # Find maximum +POST /api/v1/analytics/group-by # Group by aggregation +POST /api/v1/analytics/time-series # Time series analysis +GET /api/v1/analytics/metrics # List custom metrics +GET /api/v1/analytics/metrics/:name # Get metric value +``` + +## Dashboard Integration + +```typescript +// Define a dashboard with multiple metrics +const salesDashboard = { + title: 'Sales Dashboard', + metrics: [ + { + title: 'Total Revenue', + query: { + object: 'order', + aggregation: 'sum', + field: 'amount', + }, + }, + { + title: 'Revenue by Region', + query: { + object: 'order', + aggregations: [{ function: 'sum', field: 'amount', as: 'revenue' }], + groupBy: ['account.billing_region'], + }, + }, + ], +}; + +// Execute all dashboard queries +const dashboardData = await analytics.executeDashboard(salesDashboard); +``` + +## Advanced Features + +### Query Caching + +```typescript +// Enable caching for expensive queries +const result = await analytics.query({ + object: 'order', + aggregations: [{ function: 'sum', field: 'amount' }], + cache: { + enabled: true, + ttl: 600, // 10 minutes + }, +}); + +// Invalidate cache when data changes +analytics.invalidateCache('order'); +``` + +### Comparative Analytics + +```typescript +// Compare current vs. previous period +const comparison = await analytics.compare({ + object: 'order', + aggregation: 'sum', + field: 'amount', + currentPeriod: { + start: '2024-01-01', + end: '2024-01-31', + }, + comparisonPeriod: { + start: '2023-12-01', + end: '2023-12-31', + }, +}); + +// Result: +// { +// current: 125000, +// comparison: 110000, +// change: 15000, +// percentChange: 13.64 +// } +``` + +### Funnel Analysis + +```typescript +// Define a conversion funnel +const funnel = await analytics.funnel({ + steps: [ + { object: 'lead', stage: 'new' }, + { object: 'lead', stage: 'qualified' }, + { object: 'opportunity', stage: 'proposal' }, + { object: 'opportunity', stage: 'closed_won' }, + ], + dateRange: { + start: '2024-01-01', + end: '2024-01-31', + }, +}); + +// Result: +// { +// steps: [ +// { stage: 'new', count: 1000, percentage: 100 }, +// { stage: 'qualified', count: 450, percentage: 45 }, +// { stage: 'proposal', count: 200, percentage: 20 }, +// { stage: 'closed_won', count: 75, percentage: 7.5 }, +// ], +// overallConversion: 0.075 +// } +``` + +## Contract Implementation + +Implements `IAnalyticsService` from `@objectstack/spec/contracts`: + +```typescript +interface IAnalyticsService { + count(options: CountOptions): Promise; + sum(options: AggregationOptions): Promise; + avg(options: AggregationOptions): Promise; + min(options: AggregationOptions): Promise; + max(options: AggregationOptions): Promise; + groupBy(options: GroupByOptions): Promise; + timeSeries(options: TimeSeriesOptions): Promise; + defineMetric(metric: MetricDefinition): void; + getMetric(name: string): Promise; +} +``` + +## Performance Optimization + +1. **Choose the Right Driver**: Use SQL for large datasets, InMemory for small +2. **Enable Caching**: Cache expensive queries with appropriate TTL +3. **Optimize Filters**: Filter early to reduce dataset size +4. **Use Indexes**: Ensure database indexes on frequently queried fields +5. **Batch Queries**: Execute multiple metrics in a single dashboard query + +## Best Practices + +1. **Driver Selection**: Start with ObjectQL, optimize to SQL if needed +2. **Metric Definitions**: Define reusable metrics for consistency +3. **Cache Strategy**: Cache expensive queries, invalidate on data changes +4. **Time Series**: Use appropriate intervals (hour/day/week/month) +5. **Group By**: Limit grouping dimensions to avoid explosion of result sets + +## License + +Apache-2.0 + +## See Also + +- [@objectstack/objectql](../../objectql/) +- [@objectstack/spec/contracts](../../spec/src/contracts/) +- [Analytics Guide](/content/docs/guides/analytics/) diff --git a/packages/services/service-automation/README.md b/packages/services/service-automation/README.md new file mode 100644 index 000000000..2514f3855 --- /dev/null +++ b/packages/services/service-automation/README.md @@ -0,0 +1,437 @@ +# @objectstack/service-automation + +Automation Service for ObjectStack — implements `IAutomationService` with plugin-based DAG (Directed Acyclic Graph) flow execution engine. + +## Features + +- **Flow Execution Engine**: Execute multi-step automation flows with conditional logic +- **DAG-based Architecture**: Flows are represented as directed acyclic graphs for parallel execution +- **Trigger System**: Launch flows automatically on record changes, schedule, or manual invocation +- **Variable Management**: Pass data between flow steps with type-safe variables +- **Error Handling**: Built-in retry logic, error branches, and rollback support +- **Visual Flow Builder**: Compatible with Studio's visual flow designer +- **Type-Safe**: Full TypeScript support with flow definition validation + +## Installation + +```bash +pnpm add @objectstack/service-automation +``` + +## Basic Usage + +```typescript +import { defineStack, defineFlow } from '@objectstack/spec'; +import { ServiceAutomation } from '@objectstack/service-automation'; + +const stack = defineStack({ + services: [ServiceAutomation.configure()], +}); +``` + +## Flow Types + +ObjectStack supports three types of flows: + +### 1. Autolaunched Flows +Triggered automatically by record changes: + +```typescript +const autoFlow = defineFlow({ + name: 'welcome_email', + type: 'autolaunched', + trigger: { + object: 'user', + when: 'after_insert', + }, + steps: [ + { + type: 'action', + action: 'send_email', + inputs: { + to: '{!trigger.record.email}', + subject: 'Welcome to ObjectStack!', + body: 'Hello {!trigger.record.name}...', + }, + }, + ], +}); +``` + +### 2. Screen Flows +Interactive flows with user input: + +```typescript +const screenFlow = defineFlow({ + name: 'create_opportunity', + type: 'screen', + steps: [ + { + type: 'screen', + fields: [ + { name: 'account_id', label: 'Account', type: 'lookup', object: 'account' }, + { name: 'amount', label: 'Amount', type: 'currency' }, + { name: 'close_date', label: 'Close Date', type: 'date' }, + ], + }, + { + type: 'record_create', + object: 'opportunity', + fields: { + account_id: '{!screen.account_id}', + amount: '{!screen.amount}', + close_date: '{!screen.close_date}', + stage: 'prospecting', + }, + }, + ], +}); +``` + +### 3. Scheduled Flows +Run on a schedule (cron syntax): + +```typescript +const scheduledFlow = defineFlow({ + name: 'daily_report', + type: 'scheduled', + schedule: '0 9 * * *', // Every day at 9 AM + steps: [ + { + type: 'query', + object: 'order', + filters: [ + { field: 'created_at', operator: 'yesterday' }, + ], + output: 'orders', + }, + { + type: 'action', + action: 'send_email', + inputs: { + to: 'admin@company.com', + subject: 'Daily Orders Report', + body: 'Total orders: {!orders.length}', + }, + }, + ], +}); +``` + +## Flow Steps + +### Record Operations + +```typescript +// Create record +{ + type: 'record_create', + object: 'contact', + fields: { + name: '{!input.name}', + email: '{!input.email}', + }, + output: 'new_contact', +} + +// Update record +{ + type: 'record_update', + object: 'account', + recordId: '{!trigger.recordId}', + fields: { + status: 'active', + }, +} + +// Delete record +{ + type: 'record_delete', + object: 'task', + recordId: '{!input.taskId}', +} +``` + +### Query Step + +```typescript +{ + type: 'query', + object: 'opportunity', + filters: [ + { field: 'account_id', operator: 'eq', value: '{!trigger.record.account_id}' }, + { field: 'stage', operator: 'eq', value: 'closed_won' }, + ], + sort: [{ field: 'amount', direction: 'desc' }], + limit: 10, + output: 'opportunities', +} +``` + +### Decision (Conditional) Step + +```typescript +{ + type: 'decision', + conditions: [ + { + label: 'High Value', + expression: '{!trigger.record.amount} > 10000', + steps: [ + { type: 'action', action: 'notify_sales_manager' }, + ], + }, + { + label: 'Medium Value', + expression: '{!trigger.record.amount} > 1000', + steps: [ + { type: 'action', action: 'assign_to_sales_rep' }, + ], + }, + ], + defaultSteps: [ + { type: 'action', action: 'auto_approve' }, + ], +} +``` + +### Loop Step + +```typescript +{ + type: 'loop', + collection: '{!query_results}', + variable: 'item', + steps: [ + { + type: 'record_update', + object: 'task', + recordId: '{!item.id}', + fields: { + status: 'completed', + }, + }, + ], +} +``` + +### Custom Action Step + +```typescript +{ + type: 'action', + action: 'calculate_tax', + inputs: { + amount: '{!opportunity.amount}', + region: '{!account.billing_region}', + }, + output: 'tax_amount', +} +``` + +## Variable Expressions + +Access variables in flow steps using `{!variable.path}` syntax: + +```typescript +// Trigger record fields +'{!trigger.record.name}' +'{!trigger.record.account.industry}' + +// Screen input +'{!screen.fieldName}' + +// Query results +'{!query_results[0].name}' +'{!query_results.length}' + +// Step outputs +'{!step_name.output_field}' + +// System variables +'{!now}' +'{!today}' +'{!currentUser.id}' +``` + +## Service API + +```typescript +// Get automation service +const automation = kernel.getService('automation'); +``` + +### Execute Flow + +```typescript +// Execute a flow manually +const result = await automation.executeFlow({ + flowName: 'create_opportunity', + inputs: { + account_id: '123', + amount: 50000, + }, +}); + +// Check execution status +if (result.status === 'success') { + console.log('Flow completed:', result.outputs); +} else { + console.error('Flow failed:', result.error); +} +``` + +### Flow Management + +```typescript +// Get flow definition +const flow = await automation.getFlow('welcome_email'); + +// List all flows +const flows = await automation.listFlows(); + +// Get flow execution history +const history = await automation.getFlowHistory({ + flowName: 'daily_report', + limit: 100, +}); +``` + +### Trigger Management + +```typescript +// Register a custom trigger +automation.registerTrigger({ + name: 'on_payment_received', + description: 'Triggered when a payment is received', + async handler(context) { + // Trigger logic + return { + record: context.payment, + timestamp: new Date(), + }; + }, +}); +``` + +## REST API Endpoints + +``` +POST /api/v1/automation/flows/:name/execute # Execute flow +GET /api/v1/automation/flows # List flows +GET /api/v1/automation/flows/:name # Get flow definition +GET /api/v1/automation/flows/:name/history # Get execution history +POST /api/v1/automation/triggers/:name # Trigger a flow +``` + +## Advanced Features + +### Parallel Execution + +```typescript +const flow = defineFlow({ + name: 'parallel_processing', + steps: [ + { + type: 'parallel', + branches: [ + { + name: 'branch1', + steps: [{ type: 'action', action: 'process_a' }], + }, + { + name: 'branch2', + steps: [{ type: 'action', action: 'process_b' }], + }, + ], + }, + ], +}); +``` + +### Error Handling + +```typescript +{ + type: 'try_catch', + trySteps: [ + { type: 'action', action: 'risky_operation' }, + ], + catchSteps: [ + { + type: 'action', + action: 'send_error_notification', + inputs: { + error: '{!error.message}', + }, + }, + ], +} +``` + +### Subflows + +```typescript +{ + type: 'subflow', + flowName: 'validate_address', + inputs: { + street: '{!input.street}', + city: '{!input.city}', + }, + output: 'validated_address', +} +``` + +### Wait Step + +```typescript +{ + type: 'wait', + duration: { hours: 24 }, + nextSteps: [ + { type: 'action', action: 'send_reminder' }, + ], +} +``` + +## Best Practices + +1. **Keep Flows Simple**: Break complex logic into multiple flows +2. **Use Descriptive Names**: Name flows and steps clearly +3. **Handle Errors**: Always include error handling for critical operations +4. **Test Thoroughly**: Test flows with various input scenarios +5. **Monitor Performance**: Track flow execution times and optimize slow flows +6. **Version Control**: Store flow definitions in version control +7. **Document Intent**: Add descriptions to flows and steps + +## Performance Considerations + +- **Parallel Execution**: DAG engine automatically parallelizes independent steps +- **Batch Processing**: Use loop steps efficiently for large collections +- **Query Optimization**: Filter queries early to reduce data volume +- **Async Execution**: Long-running flows execute asynchronously + +## Contract Implementation + +Implements `IAutomationService` from `@objectstack/spec/contracts`: + +```typescript +interface IAutomationService { + executeFlow(options: FlowExecutionOptions): Promise; + getFlow(name: string): Promise; + listFlows(filter?: FlowFilter): Promise; + getFlowHistory(options: FlowHistoryOptions): Promise; + registerTrigger(trigger: TriggerDefinition): void; +} +``` + +## License + +Apache-2.0 + +## See Also + +- [@objectstack/spec/automation](../../spec/src/automation/) +- [Flow Builder Guide](/content/docs/guides/automation/) +- [Trigger Reference](/content/docs/references/automation/) diff --git a/packages/services/service-cache/README.md b/packages/services/service-cache/README.md new file mode 100644 index 000000000..5b57b6713 --- /dev/null +++ b/packages/services/service-cache/README.md @@ -0,0 +1,294 @@ +# @objectstack/service-cache + +Cache Service for ObjectStack — implements `ICacheService` with in-memory and Redis adapters. + +## Features + +- **Multiple Adapters**: In-memory (development) and Redis (production) support +- **Type-Safe**: Full TypeScript support with generic value types +- **TTL Support**: Automatic expiration with time-to-live +- **Namespace Support**: Organize cache keys by namespace +- **Pattern Matching**: Delete keys by pattern (e.g., `user:*`) +- **Statistics**: Track hit/miss rates and memory usage +- **JSON Serialization**: Automatic serialization of complex objects + +## Installation + +```bash +pnpm add @objectstack/service-cache +``` + +For Redis adapter: +```bash +pnpm add ioredis +``` + +## Basic Usage + +```typescript +import { defineStack } from '@objectstack/spec'; +import { ServiceCache } from '@objectstack/service-cache'; + +const stack = defineStack({ + services: [ + ServiceCache.configure({ + adapter: 'memory', // or 'redis' + defaultTTL: 300, // 5 minutes + }), + ], +}); +``` + +## Configuration + +### In-Memory Adapter (Development) + +```typescript +ServiceCache.configure({ + adapter: 'memory', + defaultTTL: 300, + maxSize: 1000, // Maximum number of entries +}); +``` + +### Redis Adapter (Production) + +```typescript +ServiceCache.configure({ + adapter: 'redis', + redis: { + host: 'localhost', + port: 6379, + password: process.env.REDIS_PASSWORD, + db: 0, + }, + defaultTTL: 600, +}); +``` + +## Service API + +```typescript +// Get cache service +const cache = kernel.getService('cache'); +``` + +### Set/Get Operations + +```typescript +// Set a value +await cache.set('user:123', { name: 'John', email: 'john@example.com' }); + +// Set with custom TTL (in seconds) +await cache.set('session:abc', sessionData, { ttl: 3600 }); // 1 hour + +// Get a value +const user = await cache.get('user:123'); + +// Get with type safety +const user = await cache.get('user:123'); + +// Get multiple keys +const users = await cache.mget(['user:123', 'user:456']); +``` + +### Existence & Deletion + +```typescript +// Check if key exists +const exists = await cache.has('user:123'); + +// Delete a key +await cache.del('user:123'); + +// Delete multiple keys +await cache.del(['session:abc', 'session:def']); + +// Delete by pattern +await cache.delPattern('user:*'); +``` + +### Namespaced Operations + +```typescript +// Create a namespaced cache instance +const userCache = cache.namespace('user'); + +// Set in namespace (key becomes 'user:123') +await userCache.set('123', userData); + +// Get from namespace +const user = await userCache.get('123'); + +// Clear entire namespace +await userCache.clear(); +``` + +### TTL Management + +```typescript +// Get remaining TTL (in seconds) +const ttl = await cache.ttl('session:abc'); + +// Update TTL +await cache.expire('session:abc', 7200); // 2 hours + +// Make key permanent (remove expiration) +await cache.persist('user:123'); +``` + +### Atomic Operations + +```typescript +// Increment (useful for counters) +await cache.incr('page:views:123'); // Returns new value + +// Increment by amount +await cache.incrby('score:user:123', 10); + +// Decrement +await cache.decr('inventory:product:456'); +``` + +### Batch Operations + +```typescript +// Set multiple keys at once +await cache.mset({ + 'user:123': user1Data, + 'user:456': user2Data, + 'user:789': user3Data, +}); + +// Get multiple keys +const users = await cache.mget(['user:123', 'user:456', 'user:789']); +``` + +## Advanced Features + +### Cache Aside Pattern + +```typescript +async function getUser(id: string): Promise { + // Try cache first + const cached = await cache.get(`user:${id}`); + if (cached) return cached; + + // Load from database + const user = await db.findUser(id); + + // Store in cache + await cache.set(`user:${id}`, user, { ttl: 600 }); + + return user; +} +``` + +### Cache-Through Pattern + +```typescript +async function getUserCacheThrough(id: string): Promise { + return cache.getOrSet(`user:${id}`, async () => { + return await db.findUser(id); + }, { ttl: 600 }); +} +``` + +### Invalidation on Write + +```typescript +async function updateUser(id: string, data: Partial) { + // Update database + await db.updateUser(id, data); + + // Invalidate cache + await cache.del(`user:${id}`); + + // Or update cache immediately + const updated = await db.findUser(id); + await cache.set(`user:${id}`, updated); +} +``` + +### Tagging & Invalidation + +```typescript +// Tag cache entries +await cache.set('product:123', productData, { + ttl: 600, + tags: ['products', 'category:electronics'], +}); + +// Invalidate by tag +await cache.invalidateTag('category:electronics'); +``` + +## Statistics & Monitoring + +```typescript +// Get cache statistics +const stats = await cache.stats(); +// { +// hits: 1250, +// misses: 325, +// hitRate: 0.794, +// keys: 450, +// memoryUsage: 1024000 // bytes +// } + +// Reset statistics +await cache.resetStats(); +``` + +## REST API Endpoints + +``` +GET /api/v1/cache/stats # Get cache statistics +POST /api/v1/cache/clear # Clear cache +DELETE /api/v1/cache/:key # Delete specific key +DELETE /api/v1/cache/pattern/:pattern # Delete by pattern +``` + +## Best Practices + +1. **Use Namespaces**: Organize cache keys with namespaces +2. **Set Appropriate TTLs**: Don't cache data longer than necessary +3. **Handle Misses**: Always have fallback logic when cache misses +4. **Invalidate on Write**: Clear stale cache after updates +5. **Monitor Hit Rates**: Track cache effectiveness with statistics +6. **Serialize Carefully**: Be mindful of what you serialize (avoid circular references) +7. **Use Redis in Production**: In-memory adapter is for development only + +## Performance Considerations + +- **In-Memory Adapter**: Fast but limited by server memory, not shared across instances +- **Redis Adapter**: Shared across instances, persistent, but network latency +- **TTL Strategy**: Balance between freshness and cache hit rate +- **Key Patterns**: Use consistent naming conventions for easier invalidation + +## Contract Implementation + +Implements `ICacheService` from `@objectstack/spec/contracts`: + +```typescript +interface ICacheService { + get(key: string): Promise; + set(key: string, value: T, options?: CacheOptions): Promise; + del(key: string | string[]): Promise; + has(key: string): Promise; + ttl(key: string): Promise; + expire(key: string, ttl: number): Promise; + clear(): Promise; + namespace(name: string): ICacheService; +} +``` + +## License + +Apache-2.0 + +## See Also + +- [Redis Documentation](https://redis.io/documentation) +- [@objectstack/spec/contracts](../../spec/src/contracts/) +- [Caching Best Practices](/content/docs/guides/caching/) diff --git a/packages/services/service-feed/README.md b/packages/services/service-feed/README.md new file mode 100644 index 000000000..fd233cb54 --- /dev/null +++ b/packages/services/service-feed/README.md @@ -0,0 +1,378 @@ +# @objectstack/service-feed + +Feed/Chatter Service for ObjectStack — implements `IFeedService` with in-memory adapter for comments, reactions, field changes, and record subscriptions. + +## Features + +- **Activity Feed**: Track all record changes and user activities +- **Comments & Mentions**: Add comments with @mentions support +- **Reactions**: Like, upvote, or react to records and comments +- **Field Change Tracking**: Automatic history of field value changes +- **Subscriptions**: Subscribe to records for notifications +- **Rich Content**: Support for markdown, attachments, and embeds +- **Real-time Updates**: Integrates with `@objectstack/service-realtime` for live updates + +## Installation + +```bash +pnpm add @objectstack/service-feed +``` + +## Basic Usage + +```typescript +import { defineStack } from '@objectstack/spec'; +import { ServiceFeed } from '@objectstack/service-feed'; + +const stack = defineStack({ + services: [ + ServiceFeed.configure({ + enableFieldTracking: true, + enableMentions: true, + }), + ], +}); +``` + +## Configuration + +```typescript +interface FeedServiceConfig { + /** Enable automatic field change tracking */ + enableFieldTracking?: boolean; + + /** Enable @mention support in comments */ + enableMentions?: boolean; + + /** Enable reactions (likes, upvotes, etc.) */ + enableReactions?: boolean; + + /** Maximum feed items per page */ + pageSize?: number; +} +``` + +## Service API + +```typescript +// Get feed service +const feed = kernel.getService('feed'); +``` + +### Comments + +```typescript +// Add a comment to a record +await feed.addComment({ + object: 'opportunity', + recordId: '123', + userId: 'user:456', + body: 'Great progress on this deal! @john can you follow up?', +}); + +// Get comments for a record +const comments = await feed.getComments({ + object: 'opportunity', + recordId: '123', + limit: 20, +}); + +// Update a comment +await feed.updateComment({ + commentId: 'comment:789', + body: 'Updated comment text', +}); + +// Delete a comment +await feed.deleteComment('comment:789'); +``` + +### Reactions + +```typescript +// Add a reaction +await feed.addReaction({ + targetType: 'record', // or 'comment' + targetId: '123', + userId: 'user:456', + type: 'like', // 'like', 'love', 'upvote', 'celebrate' +}); + +// Remove a reaction +await feed.removeReaction({ + targetType: 'record', + targetId: '123', + userId: 'user:456', + type: 'like', +}); + +// Get reactions for a record +const reactions = await feed.getReactions({ + targetType: 'record', + targetId: '123', +}); +// Returns: { like: 12, love: 5, upvote: 8 } +``` + +### Activity Feed + +```typescript +// Get feed for a specific record +const recordFeed = await feed.getFeed({ + object: 'account', + recordId: '123', + types: ['comment', 'field_change', 'record_created'], + limit: 50, +}); + +// Get user's personalized feed (records they follow) +const userFeed = await feed.getUserFeed({ + userId: 'user:456', + limit: 100, +}); + +// Example feed item: +// { +// id: 'feed:abc', +// type: 'field_change', +// object: 'opportunity', +// recordId: '123', +// userId: 'user:456', +// timestamp: '2024-01-15T10:30:00Z', +// data: { +// field: 'stage', +// oldValue: 'prospecting', +// newValue: 'proposal' +// } +// } +``` + +### Subscriptions + +```typescript +// Subscribe to a record +await feed.subscribe({ + object: 'opportunity', + recordId: '123', + userId: 'user:456', +}); + +// Unsubscribe from a record +await feed.unsubscribe({ + object: 'opportunity', + recordId: '123', + userId: 'user:456', +}); + +// Check if user is subscribed +const isSubscribed = await feed.isSubscribed({ + object: 'opportunity', + recordId: '123', + userId: 'user:456', +}); + +// Get all subscriptions for a user +const subscriptions = await feed.getUserSubscriptions('user:456'); +``` + +### Field Change Tracking + +Field changes are automatically tracked when `enableFieldTracking` is enabled: + +```typescript +// Automatically creates feed items like: +// { +// type: 'field_change', +// object: 'opportunity', +// recordId: '123', +// userId: 'user:456', +// data: { +// field: 'amount', +// oldValue: 50000, +// newValue: 75000 +// } +// } +``` + +Customize which fields to track: + +```typescript +ServiceFeed.configure({ + enableFieldTracking: true, + trackedObjects: { + opportunity: ['stage', 'amount', 'close_date'], + account: ['status', 'industry'], + }, +}); +``` + +## Advanced Features + +### Mentions & Notifications + +```typescript +// Parse mentions from comment body +const mentions = feed.parseMentions('Hey @john and @sarah, check this out!'); +// Returns: ['john', 'sarah'] + +// Get mentions for a user +const userMentions = await feed.getMentions('user:456', { + unreadOnly: true, +}); + +// Mark mention as read +await feed.markMentionRead({ + mentionId: 'mention:abc', + userId: 'user:456', +}); +``` + +### Rich Content + +```typescript +await feed.addComment({ + object: 'opportunity', + recordId: '123', + userId: 'user:456', + body: '## Great News!\n\nWe closed the deal at **$100k**!', + format: 'markdown', + attachments: [ + { + type: 'file', + url: 'https://example.com/contract.pdf', + name: 'contract.pdf', + size: 1024000, + }, + ], +}); +``` + +### Filtering & Search + +```typescript +// Get feed with filters +const feed = await feed.getFeed({ + object: 'opportunity', + recordId: '123', + types: ['comment'], // Only comments + userId: 'user:456', // Only from specific user + since: '2024-01-01T00:00:00Z', + until: '2024-01-31T23:59:59Z', +}); + +// Search comments +const results = await feed.searchComments({ + query: 'follow up', + object: 'opportunity', + recordId: '123', +}); +``` + +## Integration with Realtime Service + +```typescript +// Subscribe to real-time feed updates +const realtime = kernel.getService('realtime'); + +realtime.subscribe(`feed:opportunity:123`, (event) => { + if (event.type === 'comment_added') { + console.log('New comment:', event.data); + } else if (event.type === 'field_changed') { + console.log('Field changed:', event.data); + } +}); +``` + +## REST API Endpoints + +``` +POST /api/v1/feed/comments # Add comment +GET /api/v1/feed/comments/:object/:recordId # Get comments +PATCH /api/v1/feed/comments/:id # Update comment +DELETE /api/v1/feed/comments/:id # Delete comment + +POST /api/v1/feed/reactions # Add reaction +DELETE /api/v1/feed/reactions # Remove reaction +GET /api/v1/feed/reactions/:type/:id # Get reactions + +GET /api/v1/feed/:object/:recordId # Get record feed +GET /api/v1/feed/user/:userId # Get user feed + +POST /api/v1/feed/subscriptions # Subscribe +DELETE /api/v1/feed/subscriptions # Unsubscribe +GET /api/v1/feed/subscriptions/:userId # Get subscriptions +``` + +## UI Integration + +### React Hook Example + +```typescript +import { useFeed } from '@objectstack/client-react'; + +function OpportunityFeed({ recordId }: { recordId: string }) { + const { feed, addComment, loading } = useFeed({ + object: 'opportunity', + recordId, + }); + + return ( +
+ {feed.map((item) => ( + + ))} + +
+ ); +} +``` + +## Best Practices + +1. **Enable Selective Tracking**: Track only important fields to reduce noise +2. **Use Pagination**: Always paginate feed queries to avoid performance issues +3. **Subscribe Sparingly**: Don't auto-subscribe users to too many records +4. **Moderate Content**: Implement moderation for user-generated comments +5. **Archive Old Data**: Periodically archive old feed items +6. **Index Efficiently**: Ensure database indexes on object/recordId/timestamp + +## Performance Considerations + +- **In-Memory Adapter**: Current implementation is in-memory only (future: database persistence) +- **Pagination**: Always use pagination for feed queries +- **Filtering**: Filter by type and date range to reduce result set +- **Caching**: Cache recent feed items for frequently accessed records + +## Contract Implementation + +Implements `IFeedService` from `@objectstack/spec/contracts`: + +```typescript +interface IFeedService { + addComment(options: AddCommentOptions): Promise; + getComments(options: GetCommentsOptions): Promise; + updateComment(options: UpdateCommentOptions): Promise; + deleteComment(commentId: string): Promise; + + addReaction(options: AddReactionOptions): Promise; + removeReaction(options: RemoveReactionOptions): Promise; + getReactions(options: GetReactionsOptions): Promise; + + getFeed(options: GetFeedOptions): Promise; + getUserFeed(options: GetUserFeedOptions): Promise; + + subscribe(options: SubscribeOptions): Promise; + unsubscribe(options: UnsubscribeOptions): Promise; + isSubscribed(options: IsSubscribedOptions): Promise; +} +``` + +## License + +Apache-2.0 + +## See Also + +- [@objectstack/service-realtime](../service-realtime/) +- [@objectstack/spec/contracts](../../spec/src/contracts/) +- [Activity Feed Guide](/content/docs/guides/feed/) diff --git a/packages/services/service-i18n/README.md b/packages/services/service-i18n/README.md new file mode 100644 index 000000000..7604572a9 --- /dev/null +++ b/packages/services/service-i18n/README.md @@ -0,0 +1,377 @@ +# @objectstack/service-i18n + +I18n Service for ObjectStack — implements `II18nService` with file-based locale loading and translation management. + +## Features + +- **Multi-Language Support**: Manage translations for unlimited languages +- **File-Based Locales**: Load translations from JSON/YAML files +- **Namespace Support**: Organize translations by domain (e.g., `common`, `errors`, `ui`) +- **Interpolation**: Dynamic variable replacement in translations +- **Pluralization**: Language-specific plural rules +- **Fallback Chain**: Graceful fallback from dialect → base language → default +- **Type-Safe**: TypeScript support with type-safe translation keys +- **Hot Reload**: Reload translations without restarting (development) + +## Installation + +```bash +pnpm add @objectstack/service-i18n +``` + +## Basic Usage + +```typescript +import { defineStack } from '@objectstack/spec'; +import { ServiceI18n } from '@objectstack/service-i18n'; + +const stack = defineStack({ + services: [ + ServiceI18n.configure({ + defaultLocale: 'en-US', + supportedLocales: ['en-US', 'es-ES', 'fr-FR', 'de-DE'], + loadPath: './locales/{{lng}}/{{ns}}.json', + }), + ], +}); +``` + +## Configuration + +```typescript +interface I18nServiceConfig { + /** Default locale (e.g., 'en-US') */ + defaultLocale: string; + + /** List of supported locales */ + supportedLocales: string[]; + + /** Path template for locale files */ + loadPath: string; + + /** Fallback locale when translation is missing */ + fallbackLocale?: string; + + /** Enable hot reload in development */ + hotReload?: boolean; +} +``` + +## Directory Structure + +``` +locales/ +├── en-US/ +│ ├── common.json +│ ├── errors.json +│ └── ui.json +├── es-ES/ +│ ├── common.json +│ ├── errors.json +│ └── ui.json +└── fr-FR/ + ├── common.json + ├── errors.json + └── ui.json +``` + +Example `locales/en-US/common.json`: + +```json +{ + "welcome": "Welcome to ObjectStack", + "greeting": "Hello, {{name}}!", + "item_count": "You have {{count}} item", + "item_count_plural": "You have {{count}} items", + "save_button": "Save", + "cancel_button": "Cancel" +} +``` + +## Service API + +```typescript +// Get i18n service +const i18n = kernel.getService('i18n'); +``` + +### Basic Translation + +```typescript +// Simple translation +const text = await i18n.t('common:welcome'); +// "Welcome to ObjectStack" + +// With interpolation +const greeting = await i18n.t('common:greeting', { name: 'Alice' }); +// "Hello, Alice!" + +// With pluralization +const count1 = await i18n.t('common:item_count', { count: 1 }); +// "You have 1 item" + +const count5 = await i18n.t('common:item_count', { count: 5 }); +// "You have 5 items" +``` + +### Change Locale + +```typescript +// Set locale for current context +await i18n.setLocale('es-ES'); + +// Get current locale +const locale = i18n.getLocale(); +// "es-ES" + +// Translate in specific locale (without changing context) +const text = await i18n.t('common:welcome', { locale: 'fr-FR' }); +``` + +### Namespaces + +```typescript +// Load translation from 'errors' namespace +const errorMsg = await i18n.t('errors:not_found'); + +// Load multiple namespaces +await i18n.loadNamespaces(['common', 'ui', 'errors']); + +// Check if namespace is loaded +const isLoaded = i18n.isNamespaceLoaded('common'); +``` + +### Locale Management + +```typescript +// Get all supported locales +const locales = i18n.getSupportedLocales(); +// ['en-US', 'es-ES', 'fr-FR', 'de-DE'] + +// Check if locale is supported +const isSupported = i18n.isLocaleSupported('ja-JP'); +// false + +// Get locale metadata +const metadata = i18n.getLocaleMetadata('en-US'); +// { +// name: 'English (United States)', +// nativeName: 'English (United States)', +// direction: 'ltr', +// pluralRules: 'en' +// } +``` + +## Advanced Features + +### Nested Keys + +```json +{ + "user": { + "profile": { + "title": "User Profile", + "edit": "Edit Profile" + } + } +} +``` + +```typescript +await i18n.t('common:user.profile.title'); +// "User Profile" +``` + +### Arrays + +```json +{ + "days": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"] +} +``` + +```typescript +const days = await i18n.t('common:days', { returnObjects: true }); +// ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"] +``` + +### Context-Based Translations + +```json +{ + "friend": "A friend", + "friend_male": "A boyfriend", + "friend_female": "A girlfriend" +} +``` + +```typescript +await i18n.t('common:friend', { context: 'male' }); +// "A boyfriend" + +await i18n.t('common:friend', { context: 'female' }); +// "A girlfriend" +``` + +### Formatting + +```typescript +// Date formatting +const formatted = await i18n.formatDate(new Date(), { + locale: 'es-ES', + format: 'long', +}); +// "15 de enero de 2024" + +// Number formatting +const price = await i18n.formatNumber(1234.56, { + style: 'currency', + currency: 'EUR', + locale: 'fr-FR', +}); +// "1 234,56 €" + +// Relative time +const relative = await i18n.formatRelative(new Date('2024-01-01'), { + locale: 'en-US', +}); +// "3 months ago" +``` + +### Dynamic Loading + +```typescript +// Add a new locale dynamically +await i18n.addLocale('ja-JP', { + loadPath: './locales/ja-JP/{{ns}}.json', +}); + +// Remove a locale +await i18n.removeLocale('ja-JP'); + +// Reload translations (useful in development) +await i18n.reload(); +``` + +## Integration with Metadata + +Translate metadata labels automatically: + +```typescript +import { defineObject } from '@objectstack/spec'; + +const contact = defineObject({ + name: 'contact', + label: 'i18n:objects.contact.label', // References translation key + fields: [ + { + name: 'name', + label: 'i18n:fields.contact.name', + type: 'text', + }, + ], +}); + +// Translation file: locales/en-US/metadata.json +{ + "objects": { + "contact": { + "label": "Contact", + "label_plural": "Contacts" + } + }, + "fields": { + "contact": { + "name": "Full Name" + } + } +} +``` + +## REST API Endpoints + +``` +GET /api/v1/i18n/locales # Get supported locales +GET /api/v1/i18n/translations/:locale # Get all translations for locale +POST /api/v1/i18n/translate # Translate keys (batch) +``` + +## Client Integration + +### React Hook Example + +```typescript +import { useTranslation } from '@objectstack/client-react'; + +function MyComponent() { + const { t, locale, setLocale } = useTranslation(); + + return ( +
+

{t('common:welcome')}

+ +
+ ); +} +``` + +## Best Practices + +1. **Use Namespaces**: Organize translations by domain (common, ui, errors, metadata) +2. **Consistent Keys**: Use dot notation for nested keys (e.g., `user.profile.title`) +3. **Provide Context**: Use context for gender, formality, or pluralization variants +4. **Fallback Values**: Always provide fallback translations in default locale +5. **Avoid Hardcoding**: Never hardcode user-facing text; use translation keys +6. **Professional Translation**: Use professional translators for production +7. **Version Control**: Store translation files in version control + +## Locale Coverage Detection + +```typescript +// Get coverage statistics +const coverage = await i18n.getCoverage(); +// { +// 'en-US': { total: 245, missing: 0, percentage: 100 }, +// 'es-ES': { total: 245, missing: 12, percentage: 95.1 }, +// 'fr-FR': { total: 245, missing: 45, percentage: 81.6 } +// } + +// Get missing keys for a locale +const missing = await i18n.getMissingKeys('es-ES'); +// ['errors.validation.email', 'ui.dashboard.title', ...] +``` + +## Performance Considerations + +- **Lazy Loading**: Namespaces are loaded on demand +- **Caching**: Translations are cached in memory +- **Hot Reload**: Only enable in development +- **Bundle Size**: Load only required locales on client + +## Contract Implementation + +Implements `II18nService` from `@objectstack/spec/contracts`: + +```typescript +interface II18nService { + t(key: string, options?: TranslationOptions): Promise; + setLocale(locale: string): Promise; + getLocale(): string; + getSupportedLocales(): string[]; + loadNamespaces(namespaces: string[]): Promise; + formatDate(date: Date, options?: FormatOptions): Promise; + formatNumber(value: number, options?: FormatOptions): Promise; +} +``` + +## License + +Apache-2.0 + +## See Also + +- [i18next Documentation](https://www.i18next.com/) +- [@objectstack/spec/system (Translation schema)](../../spec/src/system/) +- [I18n Best Practices Guide](/content/docs/guides/i18n/) diff --git a/packages/services/service-job/README.md b/packages/services/service-job/README.md new file mode 100644 index 000000000..3afc60258 --- /dev/null +++ b/packages/services/service-job/README.md @@ -0,0 +1,371 @@ +# @objectstack/service-job + +Job Service for ObjectStack — implements `IJobService` with setInterval and cron scheduling. + +## Features + +- **Cron Scheduling**: Schedule jobs with cron expressions +- **Interval Scheduling**: Run jobs at fixed intervals +- **Job Queue**: Manage job execution queue +- **Retry Logic**: Automatic retry on failure with exponential backoff +- **Job History**: Track execution history and status +- **Concurrency Control**: Limit concurrent job execution +- **Timezone Support**: Schedule jobs in specific timezones +- **Type-Safe**: Full TypeScript support + +## Installation + +```bash +pnpm add @objectstack/service-job +``` + +## Basic Usage + +```typescript +import { defineStack } from '@objectstack/spec'; +import { ServiceJob } from '@objectstack/service-job'; + +const stack = defineStack({ + services: [ + ServiceJob.configure({ + timezone: 'America/New_York', + maxConcurrent: 5, + }), + ], +}); +``` + +## Configuration + +```typescript +interface JobServiceConfig { + /** Default timezone for cron jobs (default: 'UTC') */ + timezone?: string; + + /** Maximum concurrent job executions (default: 10) */ + maxConcurrent?: number; + + /** Enable job history tracking (default: true) */ + enableHistory?: boolean; + + /** Maximum history entries per job (default: 100) */ + maxHistorySize?: number; +} +``` + +## Service API + +```typescript +// Get job service +const jobs = kernel.getService('job'); +``` + +### Cron Jobs + +```typescript +// Schedule a job with cron expression +const job = await jobs.schedule({ + name: 'daily_report', + schedule: '0 9 * * *', // Every day at 9 AM + handler: async (context) => { + console.log('Generating daily report...'); + // Your job logic here + }, + timezone: 'America/New_York', +}); + +// Common cron patterns: +// '*/5 * * * *' - Every 5 minutes +// '0 */2 * * *' - Every 2 hours +// '0 9 * * 1-5' - Weekdays at 9 AM +// '0 0 1 * *' - First day of every month at midnight +// '0 0 * * 0' - Every Sunday at midnight +``` + +### Interval Jobs + +```typescript +// Run every 30 seconds +const job = await jobs.scheduleInterval({ + name: 'health_check', + interval: 30000, // milliseconds + handler: async (context) => { + console.log('Running health check...'); + }, +}); + +// Run every 5 minutes +const job = await jobs.scheduleInterval({ + name: 'sync_data', + interval: 5 * 60 * 1000, // 5 minutes + handler: async (context) => { + // Sync data + }, +}); +``` + +### One-Time Jobs + +```typescript +// Schedule a one-time job +const job = await jobs.scheduleOnce({ + name: 'send_reminder', + runAt: new Date('2024-12-25T09:00:00Z'), + handler: async (context) => { + console.log('Sending holiday reminder...'); + }, +}); + +// Schedule to run after a delay +const job = await jobs.scheduleOnce({ + name: 'delayed_task', + delay: 3600000, // 1 hour from now + handler: async (context) => { + console.log('Executing delayed task...'); + }, +}); +``` + +### Job Management + +```typescript +// List all jobs +const allJobs = await jobs.listJobs(); + +// Get job details +const job = await jobs.getJob('daily_report'); + +// Stop a job +await jobs.stopJob('daily_report'); + +// Resume a stopped job +await jobs.resumeJob('daily_report'); + +// Delete a job +await jobs.deleteJob('daily_report'); + +// Run a job immediately (ignoring schedule) +await jobs.runNow('daily_report'); +``` + +## Advanced Features + +### Job Context + +```typescript +const job = await jobs.schedule({ + name: 'process_orders', + schedule: '*/10 * * * *', + handler: async (context) => { + console.log('Job name:', context.jobName); + console.log('Execution ID:', context.executionId); + console.log('Scheduled time:', context.scheduledTime); + console.log('Execution count:', context.executionCount); + + // Access services + const db = context.kernel.getService('database'); + const orders = await db.find({ object: 'order', status: 'pending' }); + + // Process orders... + }, +}); +``` + +### Retry Configuration + +```typescript +const job = await jobs.schedule({ + name: 'api_sync', + schedule: '0 * * * *', // Every hour + retry: { + maxAttempts: 3, + backoff: 'exponential', // 'linear' or 'exponential' + initialDelay: 1000, // 1 second + maxDelay: 60000, // 1 minute + }, + handler: async (context) => { + // May fail and retry + await syncWithExternalAPI(); + }, +}); +``` + +### Concurrency Control + +```typescript +const job = await jobs.schedule({ + name: 'heavy_processing', + schedule: '*/5 * * * *', + concurrency: 1, // Only one instance can run at a time + handler: async (context) => { + // Long-running process + }, +}); +``` + +### Job History + +```typescript +// Get execution history for a job +const history = await jobs.getJobHistory('daily_report', { + limit: 50, + status: 'success', // 'success', 'failed', 'running' +}); + +// Example history entry: +// { +// executionId: 'exec:abc123', +// jobName: 'daily_report', +// status: 'success', +// startedAt: '2024-01-15T09:00:00Z', +// completedAt: '2024-01-15T09:05:23Z', +// duration: 323000, // milliseconds +// error: null, +// result: { records: 1250 } +// } + +// Clear history for a job +await jobs.clearHistory('daily_report'); +``` + +### Job Data & Results + +```typescript +const job = await jobs.schedule({ + name: 'data_export', + schedule: '0 0 * * *', + handler: async (context) => { + const records = await exportData(); + + // Return result data + return { + recordCount: records.length, + fileSize: calculateSize(records), + exportedAt: new Date(), + }; + }, +}); + +// Get last execution result +const lastRun = await jobs.getLastExecution('data_export'); +console.log('Last export:', lastRun.result); +``` + +## Common Patterns + +### Database Cleanup Job + +```typescript +jobs.schedule({ + name: 'cleanup_old_records', + schedule: '0 2 * * *', // 2 AM daily + handler: async (context) => { + const db = context.kernel.getService('database'); + + // Delete records older than 90 days + const cutoff = new Date(); + cutoff.setDate(cutoff.getDate() - 90); + + await db.delete({ + object: 'audit_log', + filters: [{ field: 'created_at', operator: 'lt', value: cutoff }], + }); + }, +}); +``` + +### Report Generation Job + +```typescript +jobs.schedule({ + name: 'weekly_sales_report', + schedule: '0 8 * * 1', // Mondays at 8 AM + handler: async (context) => { + const analytics = context.kernel.getService('analytics'); + + const data = await analytics.query({ + object: 'order', + aggregations: [{ function: 'sum', field: 'amount' }], + groupBy: ['sales_rep'], + filters: [{ field: 'created_at', operator: 'last_week' }], + }); + + // Generate and email report + await sendReport(data); + }, +}); +``` + +### Cache Warming Job + +```typescript +jobs.scheduleInterval({ + name: 'warm_cache', + interval: 15 * 60 * 1000, // Every 15 minutes + handler: async (context) => { + const cache = context.kernel.getService('cache'); + + // Pre-load frequently accessed data + const popularProducts = await getPopularProducts(); + await cache.set('popular_products', popularProducts, { ttl: 900 }); + }, +}); +``` + +## REST API Endpoints + +``` +GET /api/v1/jobs # List all jobs +GET /api/v1/jobs/:name # Get job details +POST /api/v1/jobs/:name/run # Run job immediately +POST /api/v1/jobs/:name/stop # Stop job +POST /api/v1/jobs/:name/resume # Resume job +DELETE /api/v1/jobs/:name # Delete job +GET /api/v1/jobs/:name/history # Get execution history +``` + +## Best Practices + +1. **Idempotent Handlers**: Job handlers should be idempotent (safe to run multiple times) +2. **Error Handling**: Always handle errors gracefully and log failures +3. **Timeout Limits**: Set reasonable timeout limits for long-running jobs +4. **Resource Limits**: Limit concurrent executions to avoid overloading the system +5. **Monitoring**: Monitor job execution times and failure rates +6. **Timezone Awareness**: Always specify timezone for cron jobs to avoid ambiguity +7. **Cleanup**: Periodically delete old job history to save storage + +## Performance Considerations + +- **Concurrency**: Limit concurrent jobs based on system resources +- **Job Duration**: Keep job execution time reasonable (< 5 minutes ideal) +- **History Size**: Limit history entries to prevent memory bloat +- **Batch Processing**: Process records in batches for large datasets + +## Contract Implementation + +Implements `IJobService` from `@objectstack/spec/contracts`: + +```typescript +interface IJobService { + schedule(options: ScheduleOptions): Promise; + scheduleInterval(options: IntervalOptions): Promise; + scheduleOnce(options: OnceOptions): Promise; + getJob(name: string): Promise; + listJobs(filter?: JobFilter): Promise; + stopJob(name: string): Promise; + resumeJob(name: string): Promise; + deleteJob(name: string): Promise; + runNow(name: string): Promise; + getJobHistory(name: string, options?: HistoryOptions): Promise; +} +``` + +## License + +Apache-2.0 + +## See Also + +- [Cron Expression Generator](https://crontab.guru/) +- [@objectstack/spec/contracts](../../spec/src/contracts/) +- [Job Scheduling Guide](/content/docs/guides/jobs/) diff --git a/packages/services/service-queue/README.md b/packages/services/service-queue/README.md new file mode 100644 index 000000000..acf770c11 --- /dev/null +++ b/packages/services/service-queue/README.md @@ -0,0 +1,453 @@ +# @objectstack/service-queue + +Queue Service for ObjectStack — implements `IQueueService` with in-memory and BullMQ adapters. + +## Features + +- **Multiple Adapters**: In-memory (development) and BullMQ/Redis (production) +- **Job Queues**: Organize work into named queues with priorities +- **Worker Pools**: Process jobs concurrently with configurable workers +- **Retry Logic**: Automatic retry with exponential backoff +- **Job Scheduling**: Delay job execution or schedule for future +- **Progress Tracking**: Track job progress and completion +- **Job Events**: Listen to job lifecycle events (active, completed, failed) +- **Rate Limiting**: Control job processing rate + +## Installation + +```bash +pnpm add @objectstack/service-queue +``` + +For BullMQ adapter (production): +```bash +pnpm add bullmq ioredis +``` + +## Basic Usage + +```typescript +import { defineStack } from '@objectstack/spec'; +import { ServiceQueue } from '@objectstack/service-queue'; + +const stack = defineStack({ + services: [ + ServiceQueue.configure({ + adapter: 'memory', // or 'bullmq' + defaultQueue: 'default', + }), + ], +}); +``` + +## Configuration + +### In-Memory Adapter (Development) + +```typescript +ServiceQueue.configure({ + adapter: 'memory', + concurrency: 5, // Max concurrent jobs +}); +``` + +### BullMQ Adapter (Production) + +```typescript +ServiceQueue.configure({ + adapter: 'bullmq', + redis: { + host: 'localhost', + port: 6379, + password: process.env.REDIS_PASSWORD, + }, + queues: { + default: { concurrency: 10 }, + email: { concurrency: 5, rateLimit: { max: 100, duration: 60000 } }, + reports: { concurrency: 2 }, + }, +}); +``` + +## Service API + +```typescript +// Get queue service +const queue = kernel.getService('queue'); +``` + +### Adding Jobs + +```typescript +// Add a simple job +await queue.add('email', 'send_welcome', { + to: 'user@example.com', + template: 'welcome', +}); + +// Add job with options +await queue.add('reports', 'generate_monthly', { + month: '2024-01', + format: 'pdf', +}, { + priority: 1, // Higher number = higher priority + attempts: 3, // Retry up to 3 times + backoff: { + type: 'exponential', + delay: 1000, + }, +}); + +// Add delayed job (runs in 1 hour) +await queue.add('notifications', 'reminder', { + userId: '123', + message: 'Don't forget!', +}, { + delay: 3600000, // 1 hour in milliseconds +}); + +// Schedule job for specific time +await queue.add('cleanup', 'old_files', {}, { + timestamp: new Date('2024-12-31T23:59:59Z').getTime(), +}); +``` + +### Processing Jobs + +```typescript +// Register a job processor +queue.process('email', async (job) => { + console.log('Processing email job:', job.data); + + // Access job data + const { to, template } = job.data; + + // Update progress + await job.updateProgress(25); + + // Send email + await sendEmail(to, template); + + await job.updateProgress(100); + + // Return result + return { sent: true, messageId: 'msg_123' }; +}); + +// Process with concurrency +queue.process('reports', 5, async (job) => { + // Up to 5 reports generated concurrently + return await generateReport(job.data); +}); + +// Process with named handler +queue.process('default', 'calculate_metrics', async (job) => { + return await calculateMetrics(job.data); +}); +``` + +### Job Management + +```typescript +// Get job by ID +const job = await queue.getJob('email', 'job_abc123'); + +// Get job status +const status = await job.getState(); +// 'waiting' | 'active' | 'completed' | 'failed' | 'delayed' + +// Remove job +await queue.removeJob('email', 'job_abc123'); + +// Retry failed job +await queue.retryJob('email', 'job_abc123'); + +// Get job result +const result = await job.returnvalue; +``` + +### Queue Operations + +```typescript +// Pause queue (stop processing new jobs) +await queue.pause('email'); + +// Resume queue +await queue.resume('email'); + +// Clear all jobs in queue +await queue.clear('email'); + +// Get queue statistics +const stats = await queue.getStats('email'); +// { +// waiting: 45, +// active: 5, +// completed: 1250, +// failed: 12, +// delayed: 3 +// } +``` + +## Advanced Features + +### Job Events + +```typescript +// Listen to job lifecycle events +queue.on('email', 'completed', async (job, result) => { + console.log(`Email sent: ${result.messageId}`); +}); + +queue.on('email', 'failed', async (job, error) => { + console.error(`Email failed: ${error.message}`); + // Send alert to admin +}); + +queue.on('email', 'progress', async (job, progress) => { + console.log(`Email progress: ${progress}%`); +}); + +queue.on('email', 'active', async (job) => { + console.log(`Email job started: ${job.id}`); +}); +``` + +### Bulk Operations + +```typescript +// Add multiple jobs at once +await queue.addBulk('email', [ + { name: 'send_welcome', data: { to: 'user1@example.com' } }, + { name: 'send_welcome', data: { to: 'user2@example.com' } }, + { name: 'send_welcome', data: { to: 'user3@example.com' } }, +]); + +// Get multiple jobs +const jobs = await queue.getJobs('email', ['waiting', 'active']); +``` + +### Job Patterns + +#### Worker Pattern + +```typescript +// Dedicated worker process +queue.process('heavy_processing', async (job) => { + // CPU-intensive work + const result = await processLargeDataset(job.data); + return result; +}); +``` + +#### Fan-Out Pattern + +```typescript +// Split work across multiple jobs +await queue.add('orchestrator', 'process_batch', { batchId: '123' }); + +queue.process('orchestrator', async (job) => { + const items = await loadBatchItems(job.data.batchId); + + // Create sub-jobs for each item + for (const item of items) { + await queue.add('worker', 'process_item', { item }); + } +}); + +queue.process('worker', async (job) => { + return await processItem(job.data.item); +}); +``` + +#### Priority Queues + +```typescript +// High priority +await queue.add('tasks', 'urgent', data, { priority: 10 }); + +// Normal priority +await queue.add('tasks', 'normal', data, { priority: 5 }); + +// Low priority +await queue.add('tasks', 'background', data, { priority: 1 }); +``` + +### Rate Limiting + +```typescript +// Limit queue to 100 jobs per minute +ServiceQueue.configure({ + adapter: 'bullmq', + queues: { + api_calls: { + concurrency: 5, + rateLimit: { + max: 100, + duration: 60000, // 1 minute + }, + }, + }, +}); +``` + +### Repeatable Jobs + +```typescript +// Add cron-based repeatable job +await queue.addRepeatable('cleanup', 'old_sessions', {}, { + cron: '0 2 * * *', // Daily at 2 AM +}); + +// Add interval-based repeatable job +await queue.addRepeatable('sync', 'data', {}, { + every: 300000, // Every 5 minutes +}); + +// Remove repeatable job +await queue.removeRepeatable('cleanup', 'old_sessions'); +``` + +## Common Use Cases + +### Email Queue + +```typescript +queue.process('email', async (job) => { + const { to, subject, body, template } = job.data; + + try { + const result = await emailProvider.send({ + to, + subject, + html: renderTemplate(template, job.data), + }); + + return { messageId: result.id, sentAt: new Date() }; + } catch (error) { + // Throw error to trigger retry + throw new Error(`Failed to send email: ${error.message}`); + } +}); + +// Add email job +await queue.add('email', 'welcome', { + to: 'newuser@example.com', + template: 'welcome', + name: 'John Doe', +}, { + attempts: 3, + backoff: { type: 'exponential', delay: 5000 }, +}); +``` + +### Report Generation + +```typescript +queue.process('reports', async (job) => { + const { reportType, userId, dateRange } = job.data; + + await job.updateProgress(10); + + // Fetch data + const data = await fetchReportData(reportType, dateRange); + + await job.updateProgress(50); + + // Generate report + const report = await generatePDF(data); + + await job.updateProgress(90); + + // Upload to storage + const url = await uploadReport(report); + + await job.updateProgress(100); + + // Notify user + await notifyUser(userId, { reportUrl: url }); + + return { url, size: report.length }; +}); +``` + +### Webhook Processing + +```typescript +queue.process('webhooks', async (job) => { + const { url, payload, headers } = job.data; + + const response = await fetch(url, { + method: 'POST', + headers, + body: JSON.stringify(payload), + }); + + if (!response.ok) { + throw new Error(`Webhook failed: ${response.status}`); + } + + return { status: response.status, responseTime: Date.now() - job.timestamp }; +}); +``` + +## REST API Endpoints + +``` +POST /api/v1/queues/:queue/jobs # Add job +GET /api/v1/queues/:queue/jobs/:id # Get job +DELETE /api/v1/queues/:queue/jobs/:id # Remove job +POST /api/v1/queues/:queue/jobs/:id/retry # Retry failed job +GET /api/v1/queues/:queue/stats # Get queue stats +POST /api/v1/queues/:queue/pause # Pause queue +POST /api/v1/queues/:queue/resume # Resume queue +DELETE /api/v1/queues/:queue # Clear queue +``` + +## Best Practices + +1. **Idempotent Jobs**: Design jobs to be safely retried +2. **Error Handling**: Always handle errors and throw to trigger retry +3. **Progress Updates**: Update progress for long-running jobs +4. **Resource Limits**: Set appropriate concurrency limits +5. **Job Data**: Keep job data small (< 1MB) +6. **Monitoring**: Track queue metrics and job failure rates +7. **Cleanup**: Remove completed jobs periodically + +## Performance Considerations + +- **Concurrency**: Tune based on system resources and external API limits +- **Rate Limiting**: Prevent overwhelming external services +- **Job Size**: Keep job payloads small for faster serialization +- **Redis Connection**: Use connection pooling for BullMQ +- **Queue Organization**: Use separate queues for different job types + +## Contract Implementation + +Implements `IQueueService` from `@objectstack/spec/contracts`: + +```typescript +interface IQueueService { + add(queue: string, name: string, data: any, options?: JobOptions): Promise; + addBulk(queue: string, jobs: JobDefinition[]): Promise; + process(queue: string, handler: JobHandler): void; + getJob(queue: string, jobId: string): Promise; + removeJob(queue: string, jobId: string): Promise; + retryJob(queue: string, jobId: string): Promise; + getStats(queue: string): Promise; + pause(queue: string): Promise; + resume(queue: string): Promise; + clear(queue: string): Promise; + on(queue: string, event: JobEvent, handler: EventHandler): void; +} +``` + +## License + +Apache-2.0 + +## See Also + +- [BullMQ Documentation](https://docs.bullmq.io/) +- [@objectstack/spec/contracts](../../spec/src/contracts/) +- [Queue Patterns Guide](/content/docs/guides/queues/) diff --git a/packages/services/service-realtime/README.md b/packages/services/service-realtime/README.md new file mode 100644 index 000000000..858e6bc2a --- /dev/null +++ b/packages/services/service-realtime/README.md @@ -0,0 +1,438 @@ +# @objectstack/service-realtime + +Realtime Service for ObjectStack — implements `IRealtimeService` with WebSocket and in-memory pub/sub. + +## Features + +- **WebSocket Support**: Real-time bidirectional communication +- **Pub/Sub Pattern**: Subscribe to channels and receive updates +- **Room-Based Architecture**: Organize connections into rooms +- **Presence Tracking**: Track online users and their status +- **Message Broadcasting**: Send messages to all connections or specific rooms +- **Event Streaming**: Stream database changes and system events +- **Auto-Reconnection**: Client auto-reconnects on connection loss +- **Type-Safe**: Full TypeScript support for events and messages + +## Installation + +```bash +pnpm add @objectstack/service-realtime +``` + +## Basic Usage + +```typescript +import { defineStack } from '@objectstack/spec'; +import { ServiceRealtime } from '@objectstack/service-realtime'; + +const stack = defineStack({ + services: [ + ServiceRealtime.configure({ + port: 3001, + path: '/ws', + }), + ], +}); +``` + +## Configuration + +```typescript +interface RealtimeServiceConfig { + /** WebSocket server port (default: 3001) */ + port?: number; + + /** WebSocket path (default: '/ws') */ + path?: string; + + /** Enable CORS (default: true) */ + cors?: boolean; + + /** Maximum connections per user (default: 10) */ + maxConnectionsPerUser?: number; + + /** Ping interval in ms (default: 30000) */ + pingInterval?: number; +} +``` + +## Service API (Server-Side) + +```typescript +// Get realtime service +const realtime = kernel.getService('realtime'); +``` + +### Broadcasting + +```typescript +// Broadcast to all connected clients +await realtime.broadcast({ + event: 'notification', + data: { message: 'System update in 5 minutes' }, +}); + +// Broadcast to specific room +await realtime.broadcastToRoom('opportunity:123', { + event: 'record_updated', + data: { recordId: '123', field: 'stage', value: 'closed_won' }, +}); + +// Broadcast to specific user +await realtime.broadcastToUser('user:456', { + event: 'mention', + data: { commentId: 'comment:789', mentionedBy: 'user:123' }, +}); +``` + +### Channel Management + +```typescript +// Join a channel (room) +await realtime.join(connectionId, 'opportunity:123'); + +// Leave a channel +await realtime.leave(connectionId, 'opportunity:123'); + +// Get all connections in a channel +const connections = await realtime.getChannelConnections('opportunity:123'); + +// Get all channels for a connection +const channels = await realtime.getConnectionChannels(connectionId); +``` + +### Presence + +```typescript +// Set user presence +await realtime.setPresence('user:456', { + status: 'online', + currentPage: '/opportunity/123', + lastActive: new Date(), +}); + +// Get user presence +const presence = await realtime.getPresence('user:456'); + +// Get all online users +const onlineUsers = await realtime.getOnlineUsers(); + +// Get users in a specific channel +const channelUsers = await realtime.getChannelPresence('opportunity:123'); +``` + +## Client-Side Usage + +### React Hook + +```typescript +import { useRealtime } from '@objectstack/client-react'; + +function OpportunityDetails({ id }: { id: string }) { + const { subscribe, send, isConnected } = useRealtime(); + + useEffect(() => { + // Subscribe to record updates + const unsubscribe = subscribe(`opportunity:${id}`, (event) => { + if (event.type === 'record_updated') { + console.log('Record updated:', event.data); + // Update UI + } + }); + + return unsubscribe; + }, [id]); + + return ( +
+ {isConnected ? '🟢 Connected' : '🔴 Disconnected'} +
+ ); +} +``` + +### JavaScript Client + +```typescript +import { RealtimeClient } from '@objectstack/client'; + +const client = new RealtimeClient({ + url: 'ws://localhost:3001/ws', + auth: { + token: 'your-auth-token', + }, +}); + +// Connect +await client.connect(); + +// Subscribe to a channel +client.subscribe('opportunity:123', (event) => { + console.log('Received event:', event); +}); + +// Send a message +client.send('typing', { + recordId: '123', + userId: 'user:456', + isTyping: true, +}); + +// Disconnect +await client.disconnect(); +``` + +## Advanced Features + +### Event Streaming + +Stream database changes in real-time: + +```typescript +// Server-side: Stream record changes +realtime.streamRecordChanges('opportunity', { + onInsert: async (record) => { + await realtime.broadcast({ + event: 'record_created', + data: { object: 'opportunity', record }, + }); + }, + onUpdate: async (record, changes) => { + await realtime.broadcastToRoom(`opportunity:${record.id}`, { + event: 'record_updated', + data: { recordId: record.id, changes }, + }); + }, + onDelete: async (recordId) => { + await realtime.broadcast({ + event: 'record_deleted', + data: { object: 'opportunity', recordId }, + }); + }, +}); +``` + +### Private Channels + +```typescript +// Server-side: Authorize private channel access +realtime.authorizeChannel = async (userId, channel) => { + if (channel.startsWith('user:')) { + // Only allow users to join their own private channel + return channel === `user:${userId}`; + } + + if (channel.startsWith('opportunity:')) { + // Check if user has access to the opportunity + const opportunityId = channel.split(':')[1]; + return await hasAccess(userId, 'opportunity', opportunityId); + } + + return false; +}; +``` + +### Typing Indicators + +```typescript +// Client sends typing event +client.send('typing', { + recordId: '123', + userId: 'user:456', + isTyping: true, +}); + +// Server broadcasts to room +realtime.on('typing', async (connectionId, data) => { + await realtime.broadcastToRoom(`opportunity:${data.recordId}`, { + event: 'user_typing', + data: { userId: data.userId, isTyping: data.isTyping }, + }, { exclude: [connectionId] }); // Don't send back to sender +}); + +// Other clients receive typing notification +client.subscribe('opportunity:123', (event) => { + if (event.type === 'user_typing') { + showTypingIndicator(event.data.userId, event.data.isTyping); + } +}); +``` + +### Live Cursor Tracking + +```typescript +// Client sends cursor position +client.send('cursor', { + recordId: '123', + x: 450, + y: 200, +}); + +// Server broadcasts to room +realtime.on('cursor', async (connectionId, data) => { + const user = await getConnectionUser(connectionId); + + await realtime.broadcastToRoom(`opportunity:${data.recordId}`, { + event: 'cursor_moved', + data: { + userId: user.id, + userName: user.name, + x: data.x, + y: data.y, + }, + }, { exclude: [connectionId] }); +}); +``` + +### Collaborative Editing + +```typescript +// Operational Transform (OT) for collaborative editing +client.send('edit', { + documentId: '123', + operation: { + type: 'insert', + position: 42, + text: 'Hello', + }, +}); + +realtime.on('edit', async (connectionId, data) => { + // Apply operation transform + const transformedOp = await applyOT(data.operation); + + // Broadcast to all editors + await realtime.broadcastToRoom(`document:${data.documentId}`, { + event: 'operation', + data: transformedOp, + }, { exclude: [connectionId] }); +}); +``` + +## Integration with ObjectStack Features + +### Feed Updates + +```typescript +// When a comment is added +feed.on('comment_added', async (comment) => { + await realtime.broadcastToRoom(`${comment.object}:${comment.recordId}`, { + event: 'feed_update', + data: { type: 'comment', comment }, + }); +}); +``` + +### Workflow Status + +```typescript +// When a flow step completes +automation.on('step_completed', async (execution) => { + await realtime.broadcastToUser(execution.userId, { + event: 'flow_progress', + data: { + flowId: execution.flowId, + step: execution.currentStep, + progress: execution.progress, + }, + }); +}); +``` + +### Analytics Dashboard + +```typescript +// Stream real-time metrics +setInterval(async () => { + const metrics = await analytics.getCurrentMetrics(); + + await realtime.broadcastToRoom('dashboard:sales', { + event: 'metrics_update', + data: metrics, + }); +}, 5000); // Every 5 seconds +``` + +## Connection Events + +```typescript +// Server-side event handlers +realtime.on('connection', async (connectionId, userId) => { + console.log(`User ${userId} connected (${connectionId})`); + + // Set initial presence + await realtime.setPresence(userId, { status: 'online' }); +}); + +realtime.on('disconnection', async (connectionId, userId) => { + console.log(`User ${userId} disconnected`); + + // Update presence + await realtime.setPresence(userId, { + status: 'offline', + lastSeen: new Date(), + }); +}); + +realtime.on('error', async (connectionId, error) => { + console.error(`Connection error:`, error); +}); +``` + +## Best Practices + +1. **Channel Organization**: Use namespaced channels (e.g., `object:recordId`) +2. **Authorization**: Always verify channel access before joining +3. **Message Size**: Keep messages small (< 10KB) +4. **Rate Limiting**: Limit message frequency per connection +5. **Cleanup**: Remove disconnected users from channels +6. **Heartbeat**: Implement ping/pong for connection health +7. **Compression**: Enable WebSocket compression for large messages + +## Performance Considerations + +- **Scaling**: Use Redis adapter for multi-server deployments +- **Connection Pooling**: Limit concurrent connections per user +- **Channel Limits**: Limit channels per connection +- **Message Batching**: Batch frequent updates to reduce traffic +- **Binary Protocol**: Use binary for large data transfers + +## REST API Endpoints + +``` +POST /api/v1/realtime/broadcast # Broadcast to all +POST /api/v1/realtime/broadcast/room/:room # Broadcast to room +POST /api/v1/realtime/broadcast/user/:userId # Broadcast to user +GET /api/v1/realtime/presence # Get online users +GET /api/v1/realtime/presence/:userId # Get user presence +GET /api/v1/realtime/channels/:channel # Get channel connections +``` + +## Contract Implementation + +Implements `IRealtimeService` from `@objectstack/spec/contracts`: + +```typescript +interface IRealtimeService { + broadcast(message: Message): Promise; + broadcastToRoom(room: string, message: Message): Promise; + broadcastToUser(userId: string, message: Message): Promise; + join(connectionId: string, channel: string): Promise; + leave(connectionId: string, channel: string): Promise; + setPresence(userId: string, presence: PresenceData): Promise; + getPresence(userId: string): Promise; + getOnlineUsers(): Promise; + on(event: string, handler: EventHandler): void; +} +``` + +## License + +Apache-2.0 + +## See Also + +- [WebSocket API Documentation](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) +- [@objectstack/client](../../client/) +- [@objectstack/client-react](../../client-react/) +- [Realtime Features Guide](/content/docs/guides/realtime/) diff --git a/packages/services/service-storage/README.md b/packages/services/service-storage/README.md new file mode 100644 index 000000000..c02b93f2d --- /dev/null +++ b/packages/services/service-storage/README.md @@ -0,0 +1,466 @@ +# @objectstack/service-storage + +Storage Service for ObjectStack — implements `IStorageService` with local filesystem and S3 adapter skeleton. + +## Features + +- **Multiple Adapters**: Local filesystem (development) and S3-compatible storage (production) +- **File Upload**: Upload files with automatic path management +- **File Download**: Retrieve files with streaming support +- **URL Generation**: Generate signed URLs for secure access +- **Metadata**: Store and retrieve file metadata +- **Directory Operations**: Create, list, and delete directories +- **Multipart Upload**: Support for large file uploads +- **Type-Safe**: Full TypeScript support + +## Installation + +```bash +pnpm add @objectstack/service-storage +``` + +For S3 adapter: +```bash +pnpm add @aws-sdk/client-s3 @aws-sdk/s3-request-presigner +``` + +## Basic Usage + +```typescript +import { defineStack } from '@objectstack/spec'; +import { ServiceStorage } from '@objectstack/service-storage'; + +const stack = defineStack({ + services: [ + ServiceStorage.configure({ + adapter: 'local', // or 's3' + basePath: './uploads', + }), + ], +}); +``` + +## Configuration + +### Local Filesystem Adapter (Development) + +```typescript +ServiceStorage.configure({ + adapter: 'local', + basePath: './uploads', + baseUrl: 'http://localhost:3000/uploads', +}); +``` + +### S3 Adapter (Production) + +```typescript +ServiceStorage.configure({ + adapter: 's3', + s3: { + bucket: 'my-bucket', + region: 'us-east-1', + credentials: { + accessKeyId: process.env.AWS_ACCESS_KEY_ID, + secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, + }, + }, + baseUrl: 'https://my-bucket.s3.amazonaws.com', +}); +``` + +### S3-Compatible Services (Cloudflare R2, DigitalOcean Spaces, MinIO) + +```typescript +ServiceStorage.configure({ + adapter: 's3', + s3: { + bucket: 'my-bucket', + region: 'auto', + endpoint: 'https://r2.cloudflarestorage.com/account-id', + credentials: { + accessKeyId: process.env.R2_ACCESS_KEY_ID, + secretAccessKey: process.env.R2_SECRET_ACCESS_KEY, + }, + }, +}); +``` + +## Service API + +```typescript +// Get storage service +const storage = kernel.getService('storage'); +``` + +### File Upload + +```typescript +// Upload a file from buffer +await storage.upload({ + path: 'documents/contract.pdf', + data: fileBuffer, + contentType: 'application/pdf', + metadata: { + userId: 'user:123', + category: 'contracts', + }, +}); + +// Upload from stream +await storage.uploadStream({ + path: 'videos/demo.mp4', + stream: fileStream, + contentType: 'video/mp4', +}); + +// Upload with automatic path generation +const path = await storage.uploadAuto({ + data: fileBuffer, + fileName: 'profile.jpg', + folder: 'avatars', + contentType: 'image/jpeg', +}); +// Returns: 'avatars/2024/01/15/abc123-profile.jpg' +``` + +### File Download + +```typescript +// Download file as buffer +const file = await storage.download('documents/contract.pdf'); +console.log(file.data); // Buffer +console.log(file.contentType); // 'application/pdf' +console.log(file.size); // File size in bytes + +// Download as stream +const stream = await storage.downloadStream('videos/demo.mp4'); +stream.pipe(res); // Pipe to HTTP response +``` + +### File Management + +```typescript +// Check if file exists +const exists = await storage.exists('documents/contract.pdf'); + +// Get file metadata +const metadata = await storage.getMetadata('documents/contract.pdf'); +// { +// size: 1024000, +// contentType: 'application/pdf', +// lastModified: Date, +// metadata: { userId: 'user:123', category: 'contracts' } +// } + +// Delete file +await storage.delete('documents/contract.pdf'); + +// Copy file +await storage.copy({ + from: 'documents/contract.pdf', + to: 'archive/2024/contract.pdf', +}); + +// Move file +await storage.move({ + from: 'temp/upload.pdf', + to: 'documents/contract.pdf', +}); +``` + +### Directory Operations + +```typescript +// List files in directory +const files = await storage.list('documents', { + recursive: false, + limit: 100, +}); +// Returns: ['contract.pdf', 'invoice.pdf', 'report.docx'] + +// List with metadata +const files = await storage.listDetailed('documents'); +// Returns: [ +// { path: 'contract.pdf', size: 1024000, lastModified: Date }, +// { path: 'invoice.pdf', size: 512000, lastModified: Date }, +// ] + +// Delete directory and all contents +await storage.deleteDirectory('temp'); +``` + +### URL Generation + +```typescript +// Generate public URL (for public files) +const url = storage.getUrl('public/logo.png'); +// 'https://my-bucket.s3.amazonaws.com/public/logo.png' + +// Generate signed URL (for private files, expires in 1 hour) +const signedUrl = await storage.getSignedUrl('documents/contract.pdf', { + expiresIn: 3600, + operation: 'read', // or 'write' +}); +// 'https://my-bucket.s3.amazonaws.com/documents/contract.pdf?X-Amz-Signature=...' + +// Generate upload URL (for direct client uploads) +const uploadUrl = await storage.getUploadUrl('uploads/temp.pdf', { + expiresIn: 900, // 15 minutes + contentType: 'application/pdf', + maxSize: 10485760, // 10MB +}); +``` + +## Advanced Features + +### Multipart Upload (Large Files) + +```typescript +// Initialize multipart upload +const uploadId = await storage.initMultipartUpload({ + path: 'large-files/video.mp4', + contentType: 'video/mp4', +}); + +// Upload parts (can be done in parallel) +const parts = []; +for (let i = 0; i < chunks.length; i++) { + const part = await storage.uploadPart({ + uploadId, + partNumber: i + 1, + data: chunks[i], + }); + parts.push(part); +} + +// Complete multipart upload +await storage.completeMultipartUpload({ + uploadId, + parts, +}); + +// Or abort if failed +await storage.abortMultipartUpload(uploadId); +``` + +### Direct Browser Upload + +```typescript +// Server: Generate presigned POST URL +const presignedPost = await storage.getPresignedPost({ + path: 'uploads/${filename}', + conditions: [ + ['content-length-range', 0, 10485760], // Max 10MB + ['starts-with', '$Content-Type', 'image/'], // Only images + ], + expiresIn: 900, // 15 minutes +}); + +// Client: Upload directly to S3 from browser +const formData = new FormData(); +Object.entries(presignedPost.fields).forEach(([key, value]) => { + formData.append(key, value); +}); +formData.append('file', file); + +await fetch(presignedPost.url, { + method: 'POST', + body: formData, +}); +``` + +### Image Processing Integration + +```typescript +// Upload original image +await storage.upload({ + path: 'images/original/photo.jpg', + data: imageBuffer, + contentType: 'image/jpeg', +}); + +// Generate and upload thumbnails +const thumbnail = await resizeImage(imageBuffer, { width: 200, height: 200 }); +await storage.upload({ + path: 'images/thumbnails/photo.jpg', + data: thumbnail, + contentType: 'image/jpeg', +}); + +const medium = await resizeImage(imageBuffer, { width: 800, height: 800 }); +await storage.upload({ + path: 'images/medium/photo.jpg', + data: medium, + contentType: 'image/jpeg', +}); +``` + +### File Attachments for Records + +```typescript +// Attach file to a record +await storage.upload({ + path: `attachments/opportunity/${opportunityId}/proposal.pdf`, + data: fileBuffer, + contentType: 'application/pdf', + metadata: { + objectType: 'opportunity', + recordId: opportunityId, + uploadedBy: 'user:123', + }, +}); + +// List attachments for a record +const attachments = await storage.list(`attachments/opportunity/${opportunityId}`); + +// Delete all attachments when record is deleted +await storage.deleteDirectory(`attachments/opportunity/${opportunityId}`); +``` + +## REST API Endpoints + +``` +POST /api/v1/storage/upload # Upload file +GET /api/v1/storage/download/:path # Download file +DELETE /api/v1/storage/:path # Delete file +GET /api/v1/storage/list # List files +POST /api/v1/storage/signed-url # Generate signed URL +POST /api/v1/storage/upload-url # Generate upload URL +GET /api/v1/storage/metadata/:path # Get file metadata +``` + +## Client Integration + +### React Component Example + +```typescript +import { useStorage } from '@objectstack/client-react'; + +function FileUploader() { + const { upload, uploading, progress } = useStorage(); + + const handleUpload = async (file: File) => { + const path = await upload({ + file, + folder: 'documents', + onProgress: (percent) => console.log(`Upload: ${percent}%`), + }); + + console.log('Uploaded to:', path); + }; + + return ( +
+ handleUpload(e.target.files[0])} /> + {uploading && } +
+ ); +} +``` + +## Common Patterns + +### User Avatar Upload + +```typescript +async function uploadAvatar(userId: string, imageFile: Buffer) { + // Upload original + const path = `avatars/${userId}/original.jpg`; + await storage.upload({ + path, + data: imageFile, + contentType: 'image/jpeg', + }); + + // Generate thumbnail + const thumbnail = await resizeImage(imageFile, { width: 128, height: 128 }); + await storage.upload({ + path: `avatars/${userId}/thumbnail.jpg`, + data: thumbnail, + contentType: 'image/jpeg', + }); + + return { + original: storage.getUrl(path), + thumbnail: storage.getUrl(`avatars/${userId}/thumbnail.jpg`), + }; +} +``` + +### Document Management + +```typescript +async function uploadDocument(doc: { + recordId: string; + file: Buffer; + fileName: string; + uploadedBy: string; +}) { + const path = `documents/${doc.recordId}/${Date.now()}-${doc.fileName}`; + + await storage.upload({ + path, + data: doc.file, + contentType: getMimeType(doc.fileName), + metadata: { + recordId: doc.recordId, + uploadedBy: doc.uploadedBy, + fileName: doc.fileName, + }, + }); + + // Create signed URL for secure download + const downloadUrl = await storage.getSignedUrl(path, { expiresIn: 86400 }); // 24 hours + + return { path, downloadUrl }; +} +``` + +## Best Practices + +1. **Path Organization**: Use hierarchical paths (e.g., `object/recordId/filename`) +2. **Content Types**: Always specify correct `contentType` +3. **Security**: Use signed URLs for private files +4. **Cleanup**: Delete files when records are deleted +5. **Validation**: Validate file types and sizes before upload +6. **Metadata**: Store useful metadata with files +7. **Backups**: Implement backup strategy for S3 buckets + +## Performance Considerations + +- **Streaming**: Use streams for large files to reduce memory usage +- **CDN**: Put CloudFront or similar CDN in front of S3 +- **Compression**: Compress files before upload when appropriate +- **Caching**: Cache file URLs and metadata +- **Multipart**: Use multipart upload for files > 5MB + +## Contract Implementation + +Implements `IStorageService` from `@objectstack/spec/contracts`: + +```typescript +interface IStorageService { + upload(options: UploadOptions): Promise; + uploadStream(options: UploadStreamOptions): Promise; + download(path: string): Promise; + downloadStream(path: string): Promise; + delete(path: string): Promise; + exists(path: string): Promise; + getMetadata(path: string): Promise; + list(path: string, options?: ListOptions): Promise; + getUrl(path: string): string; + getSignedUrl(path: string, options?: SignedUrlOptions): Promise; +} +``` + +## License + +Apache-2.0 + +## See Also + +- [AWS S3 Documentation](https://docs.aws.amazon.com/s3/) +- [Cloudflare R2 Documentation](https://developers.cloudflare.com/r2/) +- [@objectstack/spec/contracts](../../spec/src/contracts/) +- [File Upload Guide](/content/docs/guides/storage/)