From fc96ec3eb013e0810975decf042cd9fb108997b0 Mon Sep 17 00:00:00 2001 From: jellespijker Date: Fri, 26 Jun 2026 08:22:22 +0200 Subject: [PATCH 1/2] [EMB-463] Bootstrap agentic files and pre-commits for python-quality-control --- .github/copilot-instructions.md | 20 ++ .../code-reviewer.instructions.md | 5 + .../hardware-integration.instructions.md | 5 + .../pr-assistant.instructions.md | 6 + .../testing-automation.instructions.md | 5 + .pre-commit-config.yaml | 67 +++++++ .talismanrc | 6 + AGENTS.md | 184 ++++++++++++++++++ 8 files changed, 298 insertions(+) create mode 100644 .github/copilot-instructions.md create mode 100644 .github/copilot-instructions/code-reviewer.instructions.md create mode 100644 .github/copilot-instructions/hardware-integration.instructions.md create mode 100644 .github/copilot-instructions/pr-assistant.instructions.md create mode 100644 .github/copilot-instructions/testing-automation.instructions.md create mode 100644 .pre-commit-config.yaml create mode 100644 .talismanrc create mode 100644 AGENTS.md diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000..ec65bb1 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,20 @@ +# GitHub Copilot Custom Instructions for python-quality-control + +Welcome! This configuration coordinates our multi-role coding assistant system to ensure that all generated code, documentation, and tests comply with `python-quality-control`'s rigorous engineering quality standards. + +## Role-Based Personas + +Depending on the context of your query, please adopt one of our specialized development personas: + +1. **[PR Assistant](.github/copilot-instructions/pr-assistant.instructions.md):** Focuses on creating logical, small, atomic commits starting with the bracketed Jira ticket key (e.g., `[EMB-463]`) and generating structured, descriptive pull request details. +2. **[Code Reviewer](.github/copilot-instructions/code-reviewer.instructions.md):** Focuses on reviewing architectural patterns (SOLID, DRY, KISS), checking for static bugs or lints, and enforcing compact files (around 300 lines, max 400 is acceptable, but prefer smaller). +3. **[Testing Automation](.github/copilot-instructions/testing-automation.instructions.md):** Focuses on pytest async tests, Jest/C++ assertions, and non-flaky testing protocols. +4. **[Hardware Integration](.github/copilot-instructions/hardware-integration.instructions.md):** Focuses on physical/virtual hardware interaction layers, dbus interfaces, sensor loops, or direct registers context. + +--- + +## Strategic Principles + +- **Future AI Optimization:** Write clean, modular files (around 300 lines, max 400 is acceptable, but prefer smaller) with single-responsibility structures. This keeps context sizes minimal and limits token overhead. +- **Secure by Design:** Actively mitigate OWASP IoT Top 10 vulnerabilities (input sanitization, safe DBus communication paths, credential separation). +- **Experimental Guardrails:** Never commit manual tests, scratch files, or temporary test scripts. diff --git a/.github/copilot-instructions/code-reviewer.instructions.md b/.github/copilot-instructions/code-reviewer.instructions.md new file mode 100644 index 0000000..43d67a8 --- /dev/null +++ b/.github/copilot-instructions/code-reviewer.instructions.md @@ -0,0 +1,5 @@ +# Code Reviewer Persona +Focus on: +- Keeping modules compact (around 300 lines). +- SOLID design principles, DRY, and KISS. +- Proper static typing (e.g., typing hints in Python, typed interfaces in C++/QML). diff --git a/.github/copilot-instructions/hardware-integration.instructions.md b/.github/copilot-instructions/hardware-integration.instructions.md new file mode 100644 index 0000000..10ecfbb --- /dev/null +++ b/.github/copilot-instructions/hardware-integration.instructions.md @@ -0,0 +1,5 @@ +# Hardware Integration Persona +Focus on: +- Safe communication path boundaries. +- invoked inside Docker containers during commit/build pipelines. +- Real-time performance checks and timing interlocks. diff --git a/.github/copilot-instructions/pr-assistant.instructions.md b/.github/copilot-instructions/pr-assistant.instructions.md new file mode 100644 index 0000000..1ea9eb2 --- /dev/null +++ b/.github/copilot-instructions/pr-assistant.instructions.md @@ -0,0 +1,6 @@ +# PR Assistant Persona +Focus on: +- Small, single-topic, functional commits. +- Every commit starts with `[EMB-XXXX]` where XXXX is the active ticket. +- Draft PR policy checkoff list. +- Upload screenshots via `gh-image` for visual changes. diff --git a/.github/copilot-instructions/testing-automation.instructions.md b/.github/copilot-instructions/testing-automation.instructions.md new file mode 100644 index 0000000..5a7a186 --- /dev/null +++ b/.github/copilot-instructions/testing-automation.instructions.md @@ -0,0 +1,5 @@ +# Testing Automation Persona +Focus on: +- Comprehensive assertions. +- pytest linter audit assertions implementation patterns. +- Mocking external hardware, DBus services, and network endpoints cleanly to avoid flakiness. diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..4e71efd --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,67 @@ +# pre-commit configuration for python-quality-control +default_stages: [commit] + +repos: + - repo: https://github.com/thoughtworks/talisman + rev: v1.32.0 + hooks: + - id: talisman-commit + + - repo: local + hooks: + - id: check-yaml + name: Check YAML Syntax + entry: |- + python3 -c ' + import sys, yaml + for f in sys.argv[1:]: + try: yaml.safe_load(open(f)) + except Exception as e: print(f"YAML Error in {f}: {e}"); sys.exit(1) + ' + language: system + files: \.(yaml|yml)$ + + - id: check-json + name: Check JSON Syntax + entry: python3 -c 'import json, sys; [json.load(open(f)) for f in sys.argv[1:]]' + language: system + files: \.json$ + - id: check-agent-artifacts + name: Check for Agent Artifacts + entry: >- + bash -c 'if [ "$SKIP_PRE_COMMIT" = "1" ]; then exit 0; fi; for f in "$@"; do if [[ "$f" =~ (task(_.*)?\.md|task\..*|implementation_plan(_.*)?\.md|implementation_plan\..*|walkthrough(_.*)?\.md|walkthrough\..*|\.playwright-cli|__pycache__|scratch_.*|scratch\..*|temp_.*|temp\..*|test_scratch_.*|test_scratch\..*|test_agent_.*|test_agent\..*|test_temp_.*|test_temp\..*|test_run_.*|test_run\..*|test_debug_.*|test_debug\..*|test_mock_.*|test_mock\..*|test_quick_.*|test_quick\..*|test_sandbox_.*|test_sandbox\..*|test_test_.*|test_test\..*|^test_[^/]+\.(py|js|ts|tsx)$) ]]; then echo "ERROR: Agent tracking/development artifact \"$f\" must not be committed to the repository."; exit 1; fi; done' -- + language: system + - id: isort + name: Sort Imports (isort - new files only) + entry: >- + bash -c 'if [ "$SKIP_PRE_COMMIT" = "1" ]; then exit 0; fi; if ! command -v isort &>/dev/null; then echo "WARNING: isort not found. Skipping import sorting."; exit 0; fi; new_files=$(git diff --cached --diff-filter=A --name-only); args=(); for f in "$@"; do if echo "$new_files" | grep -Fx "$f" >/dev/null; then args+=("$f"); fi; done; if [ ${#args[@]} -gt 0 ]; then exec isort "${args[@]}"; fi' -- + language: system + types: [python] + + - id: black + name: Format Code (black - new files only) + entry: >- + bash -c 'if [ "$SKIP_PRE_COMMIT" = "1" ]; then exit 0; fi; if ! command -v black &>/dev/null; then echo "WARNING: black not found. Skipping code formatting."; exit 0; fi; new_files=$(git diff --cached --diff-filter=A --name-only); args=(); for f in "$@"; do if echo "$new_files" | grep -Fx "$f" >/dev/null; then args+=("$f"); fi; done; if [ ${#args[@]} -gt 0 ]; then exec black "${args[@]}"; fi' -- + language: system + types: [python] + + - id: flake8 + name: Lint Code (flake8) + entry: >- + bash -c 'if [ "$SKIP_PRE_COMMIT" = "1" ]; then exit 0; fi; if ! command -v flake8 &>/dev/null; then echo "WARNING: flake8 not found. Skipping code linting."; exit 0; fi; exec flake8 "$@"' -- + language: system + types: [python] + - id: check-local-paths + name: Check for Local Path References + entry: >- + bash -c 'if [ "$SKIP_PRE_COMMIT" = "1" ]; then exit 0; fi; if [ -n "$HOME" ] && grep -FIn "$HOME" "$@" >/dev/stderr; then echo "ERROR: Staged files contain references to your local home directory ($HOME). Please use relative paths instead."; exit 1; fi; if grep -EIn "/(home|Users)/[a-zA-Z0-9_-]+" "$@" >/dev/stderr; then echo "ERROR: Staged files contain absolute local paths (e.g., /home/USER/ or /Users/USER/). Please use relative paths instead."; exit 1; fi' -- + language: system + files: \.(py|js|jsx|ts|tsx|sh|bash|md|cpp|h|qml)$ + + - id: check-jira-ticket + name: Check Jira Ticket in Commit Message + entry: >- + bash -c 'if [ "$SKIP_PRE_COMMIT" = "1" ]; then exit 0; fi; if ! grep -Ei "^\[(EMB|CES|COL|UC|NP)-[0-9]+\] " "$1" >/dev/null; then echo "ERROR: Commit message must start with a bracketed Jira ticket prefix (e.g., [EMB-463] Description)."; exit 1; fi' -- + language: system + stages: [commit-msg] + diff --git a/.talismanrc b/.talismanrc new file mode 100644 index 0000000..37318b9 --- /dev/null +++ b/.talismanrc @@ -0,0 +1,6 @@ +fileignoreconfig: +- filename: .github/copilot-instructions.md + checksum: 3a6b1ca5f8d83df36f1fce57285d3b589f1898440dc550943c57dbf6d6adb648 +- filename: AGENTS.md + checksum: ccadb976d29d469135c76b275a76e5c885b35131b5fb776c09127c5979fd0c94 +version: "" diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..31fcf7f --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,184 @@ +# Agent Operational & Onboarding Guide (AGENTS.md) + +Welcome, AI Agent! This document defines the operational boundaries, design patterns, testing strategies, and collaborative conventions for the `python-quality-control` repository. + +As a dynamic assistant, you must adhere strictly to these principles to maintain codebase sanity and ensure future AI developers can read, modify, and build upon your work efficiently. + +--- + +## 1. Firmware Context & Core Business Value + +The **Python Quality Control Framework** is a crucial the centralized library holding formatting rules, style linters, and verification scripts used across all python firmware layers within the UltiMaker 3D printing ecosystem (specifically powering S-Line, Factor 4, and Factor 4+ series of professional/industrial 3D printers). + +### Business Value: +Unified quality gate preventing code formatting drifts and syntax errors across 15+ submodules. + +### Repository Ecosystem & Sibling Services: +Our engineering stack is modular and distributed across separate sibling repositories. When implementing changes, always align patterns and keep cross-repository dependencies synchronized: +- **[jedi-build](https://github.com/Ultimaker/jedi-build):** Central build orchestrator. Compiles target firmware images, packaging service debians. +- **[jedi-cookbook](https://github.com/Ultimaker/jedi-cookbook):** Contains build recipes for all packages in the ecosystem. +- **[python-quality-control](https://github.com/Ultimaker/python-quality-control):** Centralized quality gating, linting, and style analysis. +- **[stardust-embedded](https://github.com/Ultimaker/stardust-embedded):** The core IoT connectivity layer that connects our printers to the UltiMaker Digital Factory. +- **[okuda](https://github.com/Ultimaker/okuda):** The main touch-screen user interface (Qt/QML) which displays printer states and processes user inputs locally. +- **[opinicus](https://github.com/Ultimaker/opinicus):** The central print orchestrator daemon that coordinates all print jobs, hardware movements, and service states. + +--- + +## 2. How the Project is Used & Ecosystem + +### System Integration: +- `python-quality-control` operates as a shared test & lint utility on the printer's Linux environment. +- It interfaces with other components via invoked inside Docker containers during commit/build pipelines. +- It is deployed as part of the system update image (`.swu` update files) compiled by `jedi-build`. + +### 🔍 Dynamic AI Skill Discovery & Installation + +To gain expert domain context, succeeding AI agents should dynamically discover and install our organization's specialized engineering skills from the centralized repository `Ultimaker/UltiCortex` on branch `UC-3695_improve_code_base_for_agentic_development`. + +Install the relevant skills before generating code: +```bash +# Search for Ultimaker skills +gh skill search ultimaker --owner Ultimaker + +# Install specialized skills dynamically +gh skill install Ultimaker/UltiCortex ultimaker-firmware-developer +``` + +### Privacy, Compliance & Sovereignty: +- **Regional Data Sovereignty (GCP):** Strict separation exists between the US and the rest of the world. All US customer data is processed in US-based GCP regions, while the rest of the world is served from EU data centers. +- **ISO-27001 Audit Traceability:** All code changes must be strictly traceable. Do not make any changes without a corresponding tracking ticket in our Jira system. +- **PII & Data Protection:** Treat all Personally Identifiable Information (PII) as highly sensitive. Never print, log, or expose PII to debuggers or standard outputs. + +--- + +## 3. Work Tracking, Git & PR Habits + +### Jira Work Tracking: +- We use **Jira** to track all work. The Jira project keys are **`EMB`** (Embedded / Firmware team tasks), **`CES`** (Customer Engineering Support), or **`COL`** (Collaboration). +- **Rule:** Before starting any development task, you **must** check or ask for an active Jira ticket number (e.g., `EMB-463` or `CES-123`). +- **Branch Naming:** All feature/bugfix branches must start with the Jira ticket number, formatted as uppercase for the key and lowercase with underscores for the rest: + ```bash + EMB-463_improve_code_base_for_agentic_development + ``` +- **Confluence Links:** Refer to Confluence for specifications and architecture overviews: [Firmware Space](https://ultimaker.atlassian.net/wiki/spaces/SF/overview). + +### Git Commit Standards: +- **Bracketed Ticket Prefix**: Every Git commit title and GitHub Pull Request title **MUST** start with the bracketed Jira key: `[EMB-XXXX] `. +- **No Semantic Prefixes**: Do **NOT** use conventional/semantic commit prefix tags (such as `feat:`, `fix:`, `chore:`, `refactor:`, etc.) in commit titles or Pull Request titles. +- **Changelog Generation**: This prefixing standard is strictly required because the repository's CHANGELOG is automatically generated by GitHub directly from PR titles. We do **NOT** use or maintain a local `CHANGELOG.md` file. +- **Atomic Commits:** Keep commits small, single-topic, and functional. +- **Commit Message Standard:** Include the *why* (the reason the change was needed), the *how* (implementation details), and other peculiarities. + +### Pull Request & Review Flow: +- **Strict Branch Guardrails:** **Never** work on or merge directly into `main`, `master`, or `staging` branches. All development must occur on separate feature/bugfix branches. +- **Developer Review:** Every change made by an AI Agent must be thoroughly reviewed by a human developer. +- **GitHub Pull Request Requirements:** An official GitHub Pull Request must be opened in **DRAFT** state. It can only be considered merge-ready when: + 1. All automated GitHub Actions status checks show **green checkmarks**. + 2. All review topics, comments, and threads are completely **resolved**. +- **Empty Initiator Checklist:** Every pull request description must end with an empty checklist for the human dev who initiated the agent, confirming they reviewed the code. +- **Visual Evidence:** UI-impacting changes require visual evidence (screenshots/recordings) added to the PR description (uploaded via browser or `gh-image` tool). Do NOT commit media files directly into the repository. +- **Human Merge Only:** Under no circumstances should an AI agent attempt to merge its own Pull Request. Merging is **strictly restricted to humans**. + +### Support Portal Review & Alert Annotations: +- **Support Documentation Audit Rule:** When introducing a **new feature** or **changing existing behavior**, you **MUST** search the UltiMaker Support page: `https://support.makerbot.com/s/global-search/` and analyze if any relevant public-facing support pages are impacted. + - If support page changes are required, add a **warning block** (`> [!WARNING]`) in the PR description advising the developer to contact the support team. Outline exactly **what changed**, **why**, and **how**, citing any existing support page URLs. +- **PR Alert Annotations:** Always annotate pull request and merge descriptions with standard GitHub markdown alerts to guide the reviewer: + ```markdown + > [!NOTE] + > Useful information that users should know, even when skimming content. + + > [!WARNING] + > Urgent info that needs immediate user attention to avoid problems. + ``` + +--- + +## 4. Directory Organization & Tech-Stack Architecture + +### Tech-Stack: +- **Languages**: Python, Shell +- **Build/Build Tooling**: None - raw runner scripts +- **Core Frameworks**: flake8, pylint, pytest, vulture, mypy + +### Core Directory Layout: +- `/cfg/`: Linters rules configurations (.flake8, pycodestyle.ini). +- `/run_flake8.sh`: Shell executable triggers. + +--- + +## 5. Designing for Future AI Generated Code + +- **Decomposed File Footprints:** Keep individual modules and files as small as possible. Individual files (Python modules, C++ sources, headers, or QML) should ideally remain **around 300 lines (max 400 lines is acceptable)** to minimize token overhead and keep context-windows clean. +- **Leverage Third-Party Libraries:** Favor mature, well-maintained third-party frameworks and libraries instead of building custom code from scratch. Search NPM, PyPI, or Conan registries before implementing custom helpers. + +--- + +## 6. Local Development Environment & Setup + +### Requirements & Prerequisites: +Directly invoke linter hooks locally: `./run_flake8.sh ` + +--- + +## 7. Security & OWASP IoT Top 10 Mitigations + +Our software is deployed on industrial 3D printing equipment. We must actively mitigate the **OWASP IoT Top 10** vulnerabilities: + +1. **Weak, Guessable, or Hardcoded Credentials:** NEVER hardcode passwords, private keys, or API tokens. Our signing keys are backed in GCP Secret Manager and retrieved securely at runtime. +2. **Insecure Network Services:** Minimize listening ports. All local/network service interfaces must authenticate requests and utilize TLS/SSL where applicable. +3. **Insecure Ecosystem Interfaces:** Secure all API endpoints, DBus, and MQTT communication paths. Validate and sanitize all incoming payloads. +4. **Lack of Secure Update Mechanism:** S-Line/Factor 4 firmware updates use the SquashFS SWU format with detached GPG signatures. Never bypass the GPG signature check. +5. **Use of Insecure or Outdated Components:** Keep our Python packages, Conan packages, and debian dependencies up to date. +6. **Insufficient Privacy Protection:** Securely handle user profiles, PII, and telemetry. Never write passwords or sensitive tokens to local log files. +7. **Insecure Data Transfer and Storage:** Encrypt sensitive credentials and configurations. Use RAM-backed filesystems (`/dev/shm`) for temporary decryption targets. +8. **Lack of Device Management:** Integrate with UltiMaker Digital Factory securely. +9. **Insecure Default Settings:** Enforce safe defaults out-of-the-box. +10. **Lack of Physical Hardening:** Secure local terminal and SSH ports. Enforce strict `umssh.sh` password-less or authenticated connections. + +--- + +## 8. Companion Guides + +Consult the following guides found within the repository or ecosystem: +- **README.md**: Standard setup and compilation guidelines. +- None extra. + +--- + +## 9. Automated Pre-Commit Tooling & Closed-Loop Cycle + +To support a robust, automated closed-loop development cycle, this repository enforces automatic static quality checks using `pre-commit` before any git commit is recorded. + +### Core Architecture & Checks: +- **Fast Static Analysis:** Hooks run formatters, linters, and credentials scanning to ensure code quality and safety. +- **Commit Guardrails:** + 1. **Jira Ticket Reference:** Commit messages *must* reference a Jira ticket prefix. + 2. **Agent Artifact & Scratch File Block:** Blocks staging/committing agent-specific tracking files (`task.md`, `implementation_plan.md`, `walkthrough.md`, scratch files, temporary `test_` scripts). + 3. **Talisman Secret Scanner:** ThoughtWorks Talisman pre-commit scanner blocks committed credentials/secrets. + 4. **Local Paths Reference Blocker:** Prevents hardcoded local absolute path references (e.g., `/home//`). + +### Setup and Manual Use: +1. **Installation:** Ensure `pre-commit` is installed and set up: + ```bash + pip install pre-commit + pre-commit install --hook-type pre-commit --hook-type commit-msg + ``` +2. **Manual Execution:** Run checks manually on staged or all files: + ```bash + pre-commit run --all-files + ``` + +--- + +## 10. Visual Validation & Verification (V&V) Guidelines + +Every feature implementation, UI refinement, or bug fix **MUST** undergo a systematic Validation & Verification (V&V) process. + +### Scenarios to Test: +- **Happy Path Scenarios:** Verify standard successful workflows. Ensure that no console exceptions, network errors, or visual regressions occur. +- **Unhappy Path Scenarios:** Verify edge cases, input limits, and error handling (e.g., failed DBus bindings, offline state, empty profiles). +- **Physical Verification / Emulation:** Where applicable, use the `ultimaker-printer-ssh` skill to push the generated build output onto a networked test printer: + ```bash + ./deploy_to_printer.sh + ``` +- **PR Visual Evidence Mandate:** For visual changes (Okuda UI, griffin_html), provide viewport screenshots/recordings of both happy and unhappy paths in your PR description. From ce075381041835dece2bee57746757c9b5011c3b Mon Sep 17 00:00:00 2001 From: jellespijker Date: Fri, 26 Jun 2026 11:54:38 +0200 Subject: [PATCH 2/2] [EMB-463] Refactor check-yaml hook to use standard inline python entrypoint --- .pre-commit-config.yaml | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 4e71efd..9c6faa6 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -11,13 +11,7 @@ repos: hooks: - id: check-yaml name: Check YAML Syntax - entry: |- - python3 -c ' - import sys, yaml - for f in sys.argv[1:]: - try: yaml.safe_load(open(f)) - except Exception as e: print(f"YAML Error in {f}: {e}"); sys.exit(1) - ' + entry: python3 -c 'import sys, yaml; [yaml.safe_load(open(f)) for f in sys.argv[1:]]' language: system files: \.(yaml|yml)$