Feat/cortex agentic skills and financial audit

.agents/AGENTS.md

@@ -18,6 +18,8 @@ When the user issues any of the following commands, you must immediately load th
 | `/build-and-deploy` | [build_and_deploy_data_product](file:///.agents/skills/build_and_deploy_data_product/SKILL.md) | Syncs environment variables, identifies configuration profiles, builds Dataform models, and deploys data products. |
 | `/query-sap-ddic` | [query_sap_ddic](file:///.agents/skills/query_sap_ddic/SKILL.md) | Inspects and dumps SAP table schemas directly from replicated SAP DDIC tables in BigQuery. |
 | `/generate-er-diagram` | [generate_er_diagram](file:///.agents/skills/generate_er_diagram/SKILL.md) | Extracts schema definitions, automatically infers entity relationships via SAP field suffixes, and generates visual ERDs. |
+| `/generate-assertions` | [dataform_assertion_generator](file:///.agents/skills/dataform_assertion_generator/SKILL.md) | Inspects data products and generates automated Dataform data-integrity assertions and ledger reconciliation checks. |
+| `/generate-lookml` | [lookml_generator](file:///.agents/skills/lookml_generator/SKILL.md) | Generates production-ready Looker LookML views, dimensions, measures, and explores from Cortex data product schemas. |
 | `/create-skill` | [create_skill](file:///.agents/skills/create_skill/SKILL.md) | Evaluates overlap, scaffolds new skills or custom folder overrides, authors instructions, and validates anatomy. |
 
 ---

.agents/skills/dataform_assertion_generator/SKILL.md

@@ -0,0 +1,51 @@
+---
+name: dataform-assertion-generator
+description: Inspects Cortex Framework data products and scaffolds Dataform data quality and business integrity assertions including primary key uniqueness, non-null checks, foreign key referential integrity, and ledger reconciliation checks.
+---
+
+# Dataform Assertion Generator Skill
+
+This skill guides the inspection, design, scaffolding, and validation of automated Dataform data quality and integrity assertions for Cortex Framework V7 data products.
+
+---
+
+## Operational Rules & Quality Gates
+
+1. **Assertion Contract:** In Dataform, an assertion query is considered **failing** if it returns one or more rows. All assertion SQL queries MUST be authored such that they return zero rows when the data is valid and return violating rows when data integrity is breached.
+2. **Namespace Isolation:** Never hardcode dataset or table names. Always resolve references dynamically using `ctx.ref(moduleConfig.sources.<source_module>.datasetId, "<table_name>")` or `${ref("<table_name>")}`.
+3. **Link Integrity:** All relative markdown links referenced in this skill must resolve to real files on disk.
+4. **Custom Overrides First:** Check the `custom/` folder before generating assertions. If project-specific compliance rules or custom thresholds exist, merge and prioritize them.
+
+---
+
+## Workflow Steps
+
+### Step 1: Target Product & Key Identification
+1. Read the target data product's `manifest.yaml` and `table_settings.default.yaml` under `src/data_modules/<namespace>/<source>/products/<product_name>/`.
+2. Inspect the table definitions in `definitions/` to identify:
+   - Primary key candidate columns (e.g. `client_mandt`, `document_number_vbeln`, `item_number_posnr`).
+   - Critical non-nullable columns (e.g. `creation_date`, `currency_key`, `amount`).
+   - Foreign key relationship bounds (e.g. line items pointing to existing header records).
+   - Domain-specific financial/accounting invariants (e.g. total debits equal total credits).
+
+### Step 2: Scaffold Assertion Definitions
+Select the appropriate assertion pattern based on the target validation rule (refer to [assertion_patterns.md](references/assertion_patterns.md)):
+
+1. **Unique Key Constraint Assertion:**
+   Checks that candidate primary keys are unique across the dataset.
+2. **Non-Null Invariant Assertion:**
+   Ensures mandatory business fields are never null.
+3. **Referential Integrity (Orphan Detection) Assertion:**
+   Detects child line items that reference non-existent parent headers.
+4. **Financial Invariant (Ledger Balance) Assertion:**
+   Verifies that total ledger debits and credits balance to zero per fiscal period.
+
+### Step 3: Write Assertion Files
+Save generated assertions into `src/data_modules/<namespace>/<source>/products/<product_name>/definitions/assertions/` or the appropriate Dataform definitions directory. Use the `.sqlx` template from [assertion_template.sqlx](assets/assertion_template.sqlx).
+
+### Step 4: Verification & Validation Gate
+1. Validate the syntax of generated assertions against BigQuery ZetaSQL dialect.
+2. Run unit tests to confirm manifest and dependency integrity:
+   ```bash
+   uv run pytest tests/external/common/validation/ -q
+   ```

.agents/skills/dataform_assertion_generator/assets/assertion_template.sqlx

@@ -0,0 +1,29 @@
+config {
+  type: "assertion",
+  description: "Validates primary key uniqueness and referential integrity for the target data product"
+}
+
+/*
+  Dataform Assertion Rule:
+  Any rows returned by this query represent failing validation records.
+  A successful assertion returns 0 rows.
+*/
+
+WITH validation_errors AS (
+  SELECT
+    client_mandt,
+    document_number_vbeln,
+    COUNT(*) AS row_count
+  FROM
+    ${ref("sales_document_headers")}
+  GROUP BY
+    client_mandt,
+    document_number_vbeln
+  HAVING
+    COUNT(*) > 1
+)
+
+SELECT
+  *
+FROM
+  validation_errors

.agents/skills/dataform_assertion_generator/references/assertion_patterns.md

@@ -0,0 +1,115 @@
+# Dataform Assertion Patterns in Cortex Framework
+
+This reference document outlines the standard assertion design patterns supported in Google Cloud Cortex Framework v7.
+
+---
+
+## 1. Core Assertion Concepts
+
+In Google Cloud Dataform, an **assertion** is a query that checks for data quality issues.
+* **Pass condition:** The assertion query returns `0` rows.
+* **Fail condition:** The assertion query returns `1 or more` rows (these rows are treated as defect records).
+
+---
+
+## 2. Standard Pattern Library
+
+### Pattern A: Multi-Column Composite Unique Key
+Validates that composite primary keys (e.g., client, company code, document number, item number) are unique.
+
+```sql
+config {
+  type: "assertion"
+}
+
+SELECT
+  client_mandt,
+  company_code_bukrs,
+  accounting_document_belnr,
+  fiscal_year_gjahr,
+  line_item_buzei,
+  COUNT(*) AS duplicate_instances
+FROM
+  ${ref("universal_journal_entry_line_items")}
+GROUP BY
+  client_mandt,
+  company_code_bukrs,
+  accounting_document_belnr,
+  fiscal_year_gjahr,
+  line_item_buzei
+HAVING
+  COUNT(*) > 1
+```
+
+---
+
+### Pattern B: Non-Null Mandatory Attributes
+Validates that essential business dimensions and foreign keys are never null.
+
+```sql
+config {
+  type: "assertion"
+}
+
+SELECT
+  *
+FROM
+  ${ref("sales_document_headers")}
+WHERE
+  client_mandt IS NULL
+  OR document_number_vbeln IS NULL
+  OR creation_date_erdat IS NULL
+```
+
+---
+
+### Pattern C: Parent-Child Referential Integrity (Orphan Detection)
+Validates that child line items have a corresponding parent document.
+
+```sql
+config {
+  type: "assertion"
+}
+
+SELECT
+  item.client_mandt,
+  item.document_number_vbeln,
+  item.item_number_posnr
+FROM
+  ${ref("sales_document_items")} AS item
+LEFT JOIN
+  ${ref("sales_document_headers")} AS header
+  ON item.client_mandt = header.client_mandt
+ AND item.document_number_vbeln = header.document_number_vbeln
+WHERE
+  header.document_number_vbeln IS NULL
+```
+
+---
+
+### Pattern D: Financial Zero-Balance Ledger Reconciliation
+In double-entry bookkeeping (SAP Universal Journal `ACDOCA`), the sum of debits and credits for any posted accounting document must equal zero.
+
+```sql
+config {
+  type: "assertion"
+}
+
+SELECT
+  client_mandt,
+  company_code_bukrs,
+  accounting_document_belnr,
+  fiscal_year_gjahr,
+  ledger_rldnr,
+  ROUND(SUM(amount_in_company_code_currency_dmbtr), 2) AS ledger_imbalance
+FROM
+  ${ref("universal_journal_entry_line_items")}
+GROUP BY
+  client_mandt,
+  company_code_bukrs,
+  accounting_document_belnr,
+  fiscal_year_gjahr,
+  ledger_rldnr
+HAVING
+  ABS(ROUND(SUM(amount_in_company_code_currency_dmbtr), 2)) > 0.05
+```

.agents/skills/lookml_generator/SKILL.md

@@ -0,0 +1,45 @@
+---
+name: lookml-generator
+description: Automatically generates production-ready LookML views, dimensions, measures, and explores for Looker and BigQuery from Google Cloud Cortex Framework v7 data product annotations, manifests, and Dataform models.
+---
+
+# LookML Generator Skill
+
+This skill guides the AI assistant in transforming Google Cloud Cortex Framework v7 data products into production-ready **Looker LookML** assets (`views/*.view.lkml` and `models/*.model.lkml`).
+
+## Workflow Overview
+
+When a user requests LookML generation (or triggers `/generate-lookml`):
+
+1. **Discover Data Product Metadata:**
+   - Locate the target product under `src/data_modules/cortex/<domain>/products/<product_name>/`.
+   - Read `manifest.yaml` to identify product type, display name, and dependencies.
+   - Inspect `table_settings.default.yaml` for table names and materialization keys.
+   - Read field definitions from `annotations/<version>/<table_name>.yaml`.
+
+2. **Map Fields to LookML Dimensions:**
+   - Consult [lookml_patterns.md](references/lookml_patterns.md) for standard type mappings.
+   - Identify Primary Keys: Set `primary_key: yes` for designated grain keys (e.g. `client_mandt`, `document_number_belnr`, etc.).
+   - Identify Dates/Timestamps: Convert SAP date suffixes (`_budat`, `_bldat`, `_bedat`, `_cpudt`) into `dimension_group` blocks with standard timeframes: `[raw, date, week, month, quarter, year]`.
+   - String & Code Fields: Map to standard `type: string` with field descriptions from the YAML annotations.
+
+3. **Generate Analytical Measures:**
+   - Always include a baseline record counter:
+     ```lookml
+     measure: count {
+       type: count
+       drill_fields: [detail*]
+     }
+     ```
+   - Identify Monetary and Amount Fields (e.g. `amount`, `netwr`, `dmbtr`, `emissions`):
+     - Generate `type: sum` with `value_format_name: usd` or appropriate decimal scaling.
+     - Generate `type: average` where relevant for operational analysis.
+
+4. **Construct Star-Schema Explores:**
+   - Create or update the LookML model file (`cortex_analytics.model.lkml`).
+   - Define explores joining transaction tables (e.g. `procurement_carbon_footprint`, `financial_ledger_reconciliation`) to master data dimensions (`supplier`, `plant`, `company_code`).
+
+5. **Output Generation:**
+   - Leverage [view_template.lkml](assets/view_template.lkml) as the foundational boilerplate.
+   - Save generated view files to `looker/views/<product_name>/<table_name>.view.lkml`.
+   - Provide the user with a summary of generated LookML files and explore join graphs.

.agents/skills/lookml_generator/assets/view_template.lkml

@@ -0,0 +1,33 @@
+# LookML View Template for Cortex Data Product Tables
+view: ${VIEW_NAME} {
+  sql_table_name: `@{GCP_PROJECT_ID}.@{DATASET_ID}.${TABLE_NAME}` ;;
+
+  # --- Primary Key ---
+  dimension: pk_${PRIMARY_KEY} {
+    primary_key: yes
+    type: string
+    sql: ${PRIMARY_KEY_SQL} ;;
+    description: "Unique surrogate key identifying the record."
+  }
+
+  # --- Standard Dimensions ---
+  ${DIMENSIONS}
+
+  # --- Dimension Groups (Dates) ---
+  ${DIMENSION_GROUPS}
+
+  # --- Measures ---
+  measure: count {
+    type: count
+    drill_fields: [detail*]
+  }
+
+  ${MEASURES}
+
+  # --- Drill Set ---
+  set: detail {
+    fields: [
+      pk_${PRIMARY_KEY}
+    ]
+  }
+}

.agents/skills/lookml_generator/references/lookml_patterns.md

@@ -0,0 +1,42 @@
+# LookML Design Patterns for Cortex Framework
+
+## 1. Type Mapping Matrix
+
+| BigQuery / SAP Semantic Type | LookML Element | Example LookML Syntax |
+| :--- | :--- | :--- |
+| Primary Key | `dimension` | `primary_key: yes` |
+| Text / Code (`CHAR`, `STRING`) | `dimension` | `type: string` |
+| Integer / Quantity (`INT64`, `NUMERIC`) | `dimension` | `type: number` |
+| Boolean (`BOOLEAN`, `FLAG`) | `dimension` | `type: yesno` |
+| Date (`DATE`) | `dimension_group` | `type: time, timeframes: [date, month, quarter, year]` |
+| Timestamp (`TIMESTAMP`) | `dimension_group` | `type: time, timeframes: [raw, time, date, week, month]` |
+| Monetary Amount | `measure` | `type: sum, value_format_name: usd` |
+| Carbon Emissions ($kg CO_2e$) | `measure` | `type: sum, value_format: "#,##0.00 "kg CO2e""` |
+
+## 2. Standard View Structure
+
+```lookml
+view: procurement_carbon_footprint {
+  sql_table_name: `@{GCP_PROJECT_ID}.@{DATASET_ID}.sustainability_and_emissions_procurement_carbon_footprint` ;;
+
+  dimension: purchase_order_key {
+    primary_key: yes
+    type: string
+    sql: CONCAT(${TABLE}.client_mandt, '-', ${TABLE}.purchase_order_ebeln, '-', CAST(${TABLE}.purchase_order_item_ebelp AS STRING)) ;;
+  }
+
+  dimension_group: purchasing_document {
+    type: time
+    timeframes: [raw, date, week, month, quarter, year]
+    convert_tz: no
+    datatype: date
+    sql: ${TABLE}.purchasing_document_date_bedat ;;
+  }
+
+  measure: total_estimated_ghg_emissions {
+    type: sum
+    sql: ${TABLE}.total_estimated_ghg_emissions_kg_co2e ;;
+    value_format: "#,##0.00 \"kg CO2e\""
+  }
+}
+```

.agents/skills/using_cortex_skills/SKILL.md

@@ -30,6 +30,10 @@ Task arrives
     │
     ├── Asked to generate ER diagrams or data model graphs? ────→ generate-er-diagram
     │
+    ├── Asked to generate data quality/integrity assertions? ──→ dataform-assertion-generator
+    │
+    ├── Asked to generate Looker LookML views or explores? ────→ lookml-generator
+    │
     ├── Asked to create/scaffold a new skill or custom rules? ─→ create-skill
     │
     └── Asked to align/review model against business rules? ────→ data-modeling-standards

.github/workflows/ci.yaml

@@ -0,0 +1,53 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: CI Unit & Validation Suite
+
+on:
+  push:
+    branches:
+      - '**'
+  pull_request:
+    branches:
+      - '**'
+permissions:
+  contents: read
+
+jobs:
+  validate-and-test:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+
+      - name: Set up Python
+        uses: actions/setup-python@42375524e23c412d93fb67b49958b491fce71c38 # v5.4.0
+        with:
+          python-version: '3.13'
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@1edb52594c857e2b5b13128931090f0640537287 # v5.3.0
+        with:
+          version: "latest"
+
+      - name: Install dependencies
+        run: uv sync
+
+      - name: Run Ruff Linting
+        run: uv run ruff check src tests
+
+      - name: Run Unit & Validation Tests
+        run: uv run pytest tests/common tests/external/common -q