Built-in AI Assistant for signal/image processing and macro automation

[PierreRaybaut]

Summary

Add a built-in AI Assistant to DataLab that can converse with the user, inspect the workspace, create and process signals/images, and write/run Python macros — all through a dockable chat panel powered by an external LLM (OpenAI-compatible API).

The assistant is implemented as a self-contained subpackage datalab.aiassistant with a clean separation between:

• Provider layer — pluggable LLM backends (OpenAI-compatible HTTP, mock).
• Tool layer — declarative registry of DataLab actions exposed to the LLM (function-calling / OpenAI tools format).
• Controller — GUI-agnostic conversation loop with confirmation callback.
• GUI — Qt dock panel (AIAssistantPanel), tool-confirmation dialog, settings dialog, and a QThread worker for non-blocking LLM calls.

Motivation

DataLab already exposes a rich processing API (Sigima) and a macro system, but discovering the right operation or writing a multi-step macro requires domain knowledge. A built-in assistant lowers the barrier for new users and speeds up exploration for experts:

• "Create a sine wave at 5 Hz then compute its FFT" → 2 tool calls, no clicks.
• "Detect blobs in this image and draw ROIs around them" → one macro.
• "Which operations can I apply to a signal?" → introspect the registry.

User-facing features

• Dockable AI Assistant panel (right side, tabified with Macro Panel).
• Settings dialog to configure provider, model, API key, base URL, temperature, HTTP timeout, max iterations, auto-approval of read-only tools.
• Tool confirmation dialog with parameter preview (and macro source code preview when the assistant proposes to run a macro).
• Mock provider for offline testing without any API key (scripted replies triggered by simple keywords).
• OpenAI-compatible provider that works with:
  • OpenAI directly (https://api.openai.com/v1)
  • GitHub Models (https://models.github.ai/inference, free tier with a GitHub PAT scoped models:read)
  • Ollama, LM Studio, Azure OpenAI, etc.
• Full FR translation of all assistant strings.

Architecture

datalab/aiassistant/
├── controller.py            # AIController (GUI-agnostic conversation loop)
├── worker.py                # AIWorker(QThread)
├── providers/
│   ├── base.py              # LLMProvider ABC, ChatMessage, AssistantMessage
│   ├── openai.py            # OpenAIProvider (urllib, no extra deps)
│   └── mock.py              # MockProvider (offline scripted replies)
├── tools/
│   ├── registry.py          # Tool, ToolResult, ToolRegistry
│   └── builtin.py           # 9 built-in tools
└── widgets/
    ├── chatpanel.py         # AIAssistantPanel + _GuiBridge
    ├── settingsdialog.py    # AISettings DataSet + dialog
    └── toolconfirmdialog.py # ToolConfirmDialog

Built-in tools exposed to the LLM

Name	Read-only	Purpose
list_objects	✅	List signals/images in current panel
get_current_panel	✅	Return active panel id
get_object_info	✅	Inspect a specific object
list_available_operations	✅	Introspect Sigima processing registry
create_synthetic_signal	❌	sin / cos / gauss / noise / ramp
create_synthetic_image	❌	gauss2d / ramp / noise / checker
load_file	❌	Load a file into a panel
apply_operation	❌	Run any registered processor by name
create_and_run_macro	❌	Create then execute a Python macro

Read-only tools can be auto-approved (configurable). Mutating tools always go through the confirmation dialog.

Threading model

LLM HTTP calls run on AIWorker(QThread) to keep the UI responsive. Tool execution and the confirmation dialog must run on the GUI thread: the _GuiBridge(QObject) in the panel marshals callables from the worker back to the GUI thread via a queued signal + QSemaphore. The controller exposes an optional execute_callback so this marshalling stays out of the core conversation logic.

Configuration

Stored in DataLab INI under a new [ai] section (datalab.config.AISection):

• enabled, provider, model, api_key, base_url
• temperature, timeout, max_iterations, auto_approve_readonly

Acceptance criteria

• Provider abstraction with at least one HTTP provider (OpenAI-compatible)
• Mock provider for offline testing (no API key required)
• Tool registry with 9 built-in tools covering inspection, creation, file loading, processing and macros
• AIController with multi-turn tool-call loop, max-iteration safety cap and per-tool confirmation callback
• Dockable Qt chat panel with new-conversation, settings and send actions
• Tool-confirmation dialog with macro source preview
• Background QThread worker — UI stays responsive during LLM calls
• Tool execution + confirmation dialogs marshalled to the GUI thread (no QObject::setParent: ... different thread warnings)
• Settings persisted in DataLab INI under [ai]
• Tolerant config loading (legacy/invalid provider values fall back to default)
• French translations for all assistant strings
• Unit tests (20 tests, mocked HTTP, scripted controller — no network)
• Sphinx documentation page (doc/features/general/ai_assistant.rst)
• Menu entry View > AI Assistant with shortcut Ctrl+Shift+A
• Release note in doc/release_notes/release_1.0X.md

Out of scope (follow-ups)

• Native Anthropic / Google / Ollama-specific providers (Ollama already works via the OpenAI-compatible base URL).
• Multimodal input (sending the current plot screenshot to a vision model).
• Streaming responses (token-by-token display).
• VS Code chat-participant integration via DataLab-Kernel + MCP.
• Persistent conversation history across DataLab sessions.

Security & privacy notes

• The API key is stored in clear text in the DataLab INI file. The Settings dialog warns the user explicitly.
• Every mutating tool call requires explicit user confirmation by default.
• The system prompt instructs the LLM to never invent operation names or parameter fields, and to prefer atomic apply_operation over macros.
• No telemetry, no DataLab-side logging of conversation content.