Add AI billing + Cost Optimization ADX dashboard view to FinOps hub dashboard - #2282
Open
Arthur Silvany (arthursilvany) wants to merge 19 commits into
Open
Conversation
Add a new Microsoft.Billing/Invoices hub app that downloads Microsoft invoice PDFs from the Billing API into the ingestion container of the hub data lake. The feature is opt-in via a new "Invoices" step in the deployment wizard and supports multiple MCA/MPA billing accounts. - New app with HTTP linked service, binary datasets, orchestrator and worker pipelines, and a configurable monthly trigger - Billing accounts are read from settings.json, populated by the wizard - New Add-FinOpsHubBillingReader cmdlet to grant the Data Factory managed identity the Billing Reader role after deployment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document the AGENTS.md symlink design, first-time repo setup via Init-Repo, and the Pester 6 requirement. Correct the build-ps command and the version reference, expand the test command reference (categories, -AllTests, -RunFailed, single-test runs), extend the architecture overview with the hub namespaces and fx modules, and add changelog conventions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a30c322e-2106-4435-a1d6-18b2380197d0
Capture the approved plan and the technical design for optional AWS and Google Cloud FOCUS ingestion in FinOps hubs. Records the spike results: exportConfig.resourceId is only used as a path segment, and focuscost_1.0.json cannot be reused because 52 of its 96 mappings are Azure-specific, so per-provider schemas selected through exportConfig.dataVersion are used instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a30c322e-2106-4435-a1d6-18b2380197d0
Add focuscost_1.2-aws.json so the existing msexports ETL can process FOCUS files exported from AWS. The schema maps the 53 FOCUS columns shared with the Azure 1.2 schema plus AvailabilityZone, x_Operation, and x_ServiceCode, all of which already exist in the Costs_raw table. The 51 Azure-specific x_ columns are omitted because AWS exports do not contain them. Validated against a real AWS FOCUS export. x_Discounts and the three PricingCurrency columns are omitted because they have no equivalent in the Data Explorer schema; adding them would require changing the raw and final tables for all providers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a30c322e-2106-4435-a1d6-18b2380197d0
Record the S3 layout, the manifest delivery contract, and three design consequences: the AWS Manifest.json must never be copied into msexports, reading it is mandatory to avoid duplicating daily refresh folders in create-new mode, and it doubles as the completion signal. Also document how ingestionId and a stable destination folder make daily refreshes replace rather than accumulate data. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a30c322e-2106-4435-a1d6-18b2380197d0
The Billing download API is a long-running operation that returns 202 with a Location header, but the Web activity did not poll it. The pipeline used the first response as if it were the final download URL, so invoices were saved empty or the copy failed outright. ADF does not allow an Until activity nested inside a ForEach, so per-invoice download moved to a new invoices_DownloadInvoiceFile child pipeline. This also gives each iteration its own variable scope, which the shared parent variables could not provide when running in parallel. - Add invoices_DownloadInvoiceFile with an Until loop that polls the operation for up to 30 minutes, falling back to Azure-AsyncOperation when Location is absent, and a Fail activity that reports InvoiceDownloadUrlNotAvailable instead of silently writing an empty file. - Mark every activity that handles the SAS URL as secureInput/secureOutput. - Guard the optional billingProfileId and purchaseOrderNumber properties with contains(). coalesce() cannot rescue a missing property because dereferencing an absent property fails to evaluate first. - Move the invoice download changelog entry from the released v15 section to Unreleased, since the feature has not shipped yet. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both design documents were written in Portuguese. Repository files must be in English, so this rewrites them in full while preserving the technical content. Also records findings from validating the AWS path end to end against a deployed v14 hub using a real customer FOCUS export: - Add section 11 documenting the run. Load Schema Mappings resolved focuscost_1.2-aws.json and Convert to Parquet succeeded against the real 19,827-row AWS parquet, so the schema needs no ETL changes. Tags and SkuPriceDetails survive the TabularTranslator as JSON and ADX materializes them as dynamic, which closes the last open type risk. - Document that the ADX drop-by tag is case-sensitive and carries the full blob path including the file name. Because the ETL lowercases the scope, exportConfig.resourceId must always be emitted in lowercase or a refresh duplicates the month silently. Ingesting the same file through both routes produced 39,654 rows and double the cost with no error raised. - Correct section 2.2, which showed FocusCost/... as the destination path. focuscost maps to the Costs hub dataset, so the path is Costs/.... - Fix three broken references to files/design-multicloud-focus.md in the plan and flag its spike results as superseded, since they still claimed FOCUS 1.0 and provenance stamping via additionalColumns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Billing download API is a long-running operation that returns 202 with a Location header, but the Web activity did not poll it. The pipeline used the first response as if it were the final download URL, so invoices were saved empty or the copy failed outright. ADF does not allow an Until activity nested inside a ForEach, so per-invoice download moved to a new invoices_DownloadInvoiceFile child pipeline. This also gives each iteration its own variable scope, which the shared parent variables could not provide when running in parallel. - Add invoices_DownloadInvoiceFile with an Until loop that polls the operation for up to 30 minutes, falling back to Azure-AsyncOperation when Location is absent, and a Fail activity that reports InvoiceDownloadUrlNotAvailable instead of silently writing an empty file. - Mark every activity that handles the SAS URL as secureInput/secureOutput. - Guard the optional billingProfileId and purchaseOrderNumber properties with contains(). coalesce() cannot rescue a missing property because dereferencing an absent property fails to evaluate first. - Move the invoice download changelog entry from the released v15 section to Unreleased, since the feature has not shipped yet. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 79763cf1-bdec-4a16-9c6a-6854ebdd3176
… assignment Data Factory rejects a container activity inside another container activity at run time, so the ForEach nested in the If Condition branch of invoices_DownloadInvoices failed with "Container activity cannot include another container activity". The pipeline deployed successfully, so the failure only surfaced on the monthly run. Replace the If Condition with two top-level ForEach activities that each iterate over an empty array when they do not apply. Billing account scopes are not part of Azure RBAC, so New-AzRoleAssignment cannot grant Billing Reader at /providers/Microsoft.Billing/billingAccounts/<id>. Use the billing role assignment API instead, matching the pattern already used by Add-FinOpsServicePrincipal, and look up the reader role definition by name so the command works across agreement types. Update the README to point at the portal instead of the equivalent az role assignment create command. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2331b11e-b4d8-460b-981d-b69dfd5eb133
Add an optional AWS connector that copies FOCUS 1.2 cost and usage exports from an Amazon S3 bucket into the hub data lake on a daily schedule, so AWS costs flow through the existing msexports ETL and land in ADX alongside Microsoft Cloud costs. Collection is driven by the AWS Data Exports manifest (dataFiles array) rather than by listing the data folder, because in "create new" mode the folder accumulates one subfolder per daily refresh and listing would duplicate the month's costs. - Add Microsoft.FinOpsHubs/AmazonWebServices app and metadata modules - Wire the module into hub.bicep and main.bicep behind enableAwsFocusIngestion (no-op when disabled) - Add a Multicloud step to createUiDefinition.json - Add AWS parameters to Deploy-FinOpsHub, gated on hub version 15.0 - Add unit tests, module README, and template/changelog docs - Correct section 5 of the multicloud FOCUS design to match the manifest-driven flow validated during the spike Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2331b11e-b4d8-460b-981d-b69dfd5eb133
Both settings compile and deploy cleanly but fail once Data Factory actually uses them. They were found by deploying the connector to a live hub and running the collection end to end. - Add the zone designator to the daily trigger start time. With timeZone UTC, Data Factory requires 'yyyy-MM-ddTHH:mm:ssZ'. Without it the trigger deploys but fails to start with InvalidWorkflowTriggerRecurrence, which aborts the hub deployment in the start-triggers script and leaves every trigger stopped. - Drop quoteAllText from the manifest sink. Data Factory rejects false with DelimitedTextInvalidSettings. The empty quoteChar on the dataset already disables quoting, so the manifest is written as raw JSON. Add a lint rule covering both traps across the template tree. Validated end to end against a deployed hub: a real AWS FOCUS 1.2 export staged with a generated manifest ingested 19,827 rows and USD 82,246.63 into Data Explorer with ProviderName AWS, matching the source file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2331b11e-b4d8-460b-981d-b69dfd5eb133
- Updated Multicloud-FOCUS-plan.md to outline the scope and rationale for optional FOCUS data ingestion from AWS and Google during FinOps hub installation. - Added new parameters for AWS and Google FOCUS ingestion in the main template and portal UI. - Enhanced security by using Key Vault for storing AWS/Google credentials. - Extended Data Factory with new Bicep modules for downloading invoices, including a new pipeline for downloading individual invoice files. - Improved error handling and validation in the invoice download process. - Updated README.md for Invoices module to reflect new pipeline and error handling scenarios.
…e to managed identity
…ashboard Integrates the AI billing page into the current FinOps hub ADX dashboard and keeps the existing Usage optimization recommendations view. Also aligns AI visuals with shared cost base queries and global billing currency filtering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d854c6ea-51ff-49f7-8b9f-1d75f702cfac
microsoft-github-policy-service
Bot
added
the
Needs: Review 👀
microsoft-github-policy-service
Bot
assigned
Michael Flanakin (flanakin) and unassigned
Michael Flanakin (flanakin)
Copilot started reviewing on behalf of
Arthur Silvany (arthursilvany)
August 26, 2026 11:08
View session
Contributor
There was a problem hiding this comment.
Pull request overview
This PR substantially expands FinOps hub capabilities by adding new Azure Data Explorer (ADX) dashboard views (Usage optimization + AI billing) and introducing two optional hub apps: Microsoft invoice download and AWS FOCUS ingestion, wired end-to-end through the hub template, portal deployment UX, docs, and PowerShell tooling.
Changes:
- Added ADX dashboard pages/queries for Usage optimization (recommendations) and AI billing.
- Added optional Invoices app (monthly Microsoft invoice download) and optional AWS FOCUS ingestion app (daily S3 collection + manifest-driven ingestion).
- Updated deployment UX/docs/PowerShell/tests to surface and validate the new options.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Core/Copy-FileToAzureBlob.ps1 | Extends settings generation/upload to include invoices.billingAccounts driven by an env var. |
| src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Core/app.bicep | Adds invoiceBillingAccounts parameter and passes it into the settings upload deployment script. |
| src/templates/finops-hub/modules/Microsoft.FinOpsHubs/AmazonWebServices/README.md | New documentation for AWS FOCUS collection behavior, requirements, and troubleshooting. |
| src/templates/finops-hub/modules/Microsoft.FinOpsHubs/AmazonWebServices/metadata.bicep | Defines the metadata contract (datasets/linked services/pipelines) for the AWS app. |
| src/templates/finops-hub/modules/Microsoft.FinOpsHubs/AmazonWebServices/app.bicep | New AWS S3 connector: linked service, datasets, pipelines, and schedule trigger. |
| src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/schemas/focuscost_1.2-aws.json | Adds an AWS-specific FOCUS 1.2 schema mapping file for the existing ETL. |
| src/templates/finops-hub/modules/Microsoft.CostManagement/Exports/app.bicep | Publishes the new AWS schema into the hub’s schema storage. |
| src/templates/finops-hub/modules/Microsoft.Billing/Invoices/README.md | New documentation for invoice download, required permissions, and validation steps. |
| src/templates/finops-hub/modules/Microsoft.Billing/Invoices/metadata.bicep | Defines the metadata contract for the Invoices app. |
| src/templates/finops-hub/modules/Microsoft.Billing/Invoices/app.bicep | New invoices downloader: Billing API polling, copy to data lake, and monthly trigger. |
| src/templates/finops-hub/modules/hub.bicep | Adds parameters and conditionally deploys the Invoices and AWS apps; updates telemetry and trigger startup dependencies. |
| src/templates/finops-hub/main.bicep | Adds top-level parameters, parses invoice billing accounts, and wires values into modules/hub.bicep. |
| src/templates/finops-hub/dashboard.json | Adds new ADX dashboard tiles/pages and KQL queries for optimization and AI billing views. |
| src/templates/finops-hub/createUiDefinition.json | Adds new wizard steps for Invoices and Multicloud (AWS) and wires outputs to template params. |
| src/templates/finops-hub/.build.config | Excludes new module READMEs from template packaging. |
| src/scripts/Deploy-Hub.ps1 | Adds switches/params to enable invoice download and pass billing account IDs. |
| src/powershell/Tests/Unit/Deploy-FinOpsHub.Tests.ps1 | Adds unit coverage around AWS FOCUS parameter passing and validation behavior. |
| src/powershell/Tests/Unit/Add-FinOpsHubResourceGraphReader.Tests.ps1 | Adds unit tests for the new Resource Graph Reader role assignment cmdlet. |
| src/powershell/Tests/Lint/DataFactorySettings.Tests.ps1 | Adds lint rules for ADF trigger startTime/timeZone and quoteAllText runtime pitfalls. |
| src/powershell/Public/Deploy-FinOpsHub.ps1 | Adds public cmdlet parameters/wiring for AWS FOCUS ingestion deployment. |
| src/powershell/Public/Add-FinOpsHubResourceGraphReader.ps1 | New helper cmdlet to grant Reader at subscription/mgmt-group scopes for ARG queries. |
| src/powershell/Public/Add-FinOpsHubBillingReader.ps1 | New helper cmdlet to grant Billing Reader at billing account scope via Billing API. |
| src/powershell/en-US/FinOpsToolkit.strings.psd1 | Adds localized strings for the new role-assignment cmdlets. |
| docs-wiki/design/Multicloud-FOCUS-plan.md | Design plan describing the optional multicloud ingestion direction and rationale. |
| docs-wiki/design/Multicloud-FOCUS-design.md | Detailed technical design and validation notes for the multicloud connector approach. |
| docs-mslearn/toolkit/hubs/template.md | Documents new template parameters (invoices + AWS FOCUS). |
| docs-mslearn/toolkit/hubs/configure-recommendations.md | Adds guidance for using Add-FinOpsHubResourceGraphReader. |
| docs-mslearn/toolkit/changelog.md | Updates changelog with new hubs features and cmdlets. |
| AGENTS.md | Enhances repo agent guidance (setup, testing, structure, and changelog conventions). |
Suppressed comments (3)
src/templates/finops-hub/dashboard.json:3923
- This AI billing trend query uses case-sensitive equality (
==) forServiceCategory/ServiceName. Using case-insensitive equality (=~) is safer and consistent with the dashboard's general KQL filtering approach.
"text": "CostsByDay\n| where (ServiceCategory == 'AI and Machine Learning' or ServiceName == 'Azure AI Search')\n| summarize EffectiveCost = round(sum(EffectiveCost), 2) by bin(ChargePeriodStart, 1d)\n| order by ChargePeriodStart asc",
src/templates/finops-hub/dashboard.json:3934
- This AI-by-service query uses case-sensitive equality (
==) forServiceCategory/ServiceName. Consider switching to=~to avoid missing rows due to casing differences.
"text": "let aiCosts = CostsByMonth | where (ServiceCategory == 'AI and Machine Learning' or ServiceName == 'Azure AI Search');\nlet totalAI = toscalar(aiCosts | summarize sum(EffectiveCost));\naiCosts\n| summarize\n EffectiveCost = round(sum(EffectiveCost), 2),\n ResourceCount = dcount(ResourceName)\n by ServiceName\n| extend SharePercent = iff(totalAI > 0, round(EffectiveCost / totalAI * 100, 1), 0.0)\n| order by EffectiveCost desc",
src/templates/finops-hub/dashboard.json:3945
- This Top resources query uses case-sensitive equality (
==) forServiceCategory/ServiceName. Using=~avoids case-related misses and aligns with the dashboard's other KQL filters.
"text": "CostsByDay\n| where (ServiceCategory == 'AI and Machine Learning' or ServiceName == 'Azure AI Search')\n| summarize\n TotalCost = round(sum(EffectiveCost), 2),\n DailyAvg = round(avg(EffectiveCost), 2)\n by ResourceName, ServiceName\n| top 5 by TotalCost desc",
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+402
to
+406
| //------------------------------------------------------------------------------ | ||
| // Invoices | ||
| //------------------------------------------------------------------------------ | ||
|
|
||
| module invoices 'Microsoft.Billing/Invoices/app.bicep' = if (enableInvoiceDownload) { |
| author: flanakin | ||
| ms.author: micflan | ||
| ms.date: 06/03/2026 | ||
| ms.date: 08/24/2026 |
| | **awsBucketName** | String | Optional. Name of the Amazon S3 bucket that contains the FOCUS export. Requires enableAwsFocusIngestion. | | | ||
| | **awsBucketPath** | String | Optional. Path to the FOCUS export root folder in the bucket. This is the folder that contains the `data` and `metadata` subfolders created by AWS Data Exports. Requires enableAwsFocusIngestion. | | | ||
| | **awsAccountId** | String | Optional. AWS account ID (management or payer account) used to identify the source scope in the data lake. Requires enableAwsFocusIngestion. | | | ||
| | **awsRegion** | String | Optional. AWS region of the S3 bucket. Requires enableAwsFocusIngestion. | "us-east-1" | |
| "kind": "inline", | ||
| "dataSourceId": "23540be2-ffc9-4b61-8c4c-05e493e682a6" | ||
| }, | ||
| "text": "let _isAI = (svcCat: string, svcName: string) { svcCat == 'AI and Machine Learning' or svcName == 'Azure AI Search' };\nlet aiCosts = CostsByDay | where _isAI(ServiceCategory, ServiceName);\nlet totalAI = toscalar(aiCosts | summarize round(sum(EffectiveCost), 2));\nlet totalCloud = toscalar(CostsByDay | summarize sum(EffectiveCost));\nlet aiPercent = iff(totalCloud > 0, round(totalAI / totalCloud * 100, 1), 0.0);\nlet resourceCount = toscalar(aiCosts | summarize dcount(ResourceName));\nlet currentMonthStart = startofmonth(now());\nlet priorMonthStart = startofmonth(datetime_add('month', -1, now()));\nlet currentMonthAI = toscalar(\n CostsPlus\n | where (ServiceCategory == 'AI and Machine Learning' or ServiceName == 'Azure AI Search')\n | where ChargePeriodStart >= currentMonthStart\n | summarize sum(EffectiveCost)\n);\nlet priorMonthAI = toscalar(\n CostsPlus\n | where (ServiceCategory == 'AI and Machine Learning' or ServiceName == 'Azure AI Search')\n | where ChargePeriodStart >= priorMonthStart and ChargePeriodStart < currentMonthStart\n | summarize sum(EffectiveCost)\n);\nlet momChange = iff(priorMonthAI > 0, round((currentMonthAI - priorMonthAI) / priorMonthAI * 100, 1), 0.0);\nprint\n ['Total AI Cost'] = totalAI,\n ['AI % of Cloud'] = aiPercent,\n ['MoM Change %'] = momChange,\n ['AI Resources'] = resourceCount", |
…ling-opt-dashboard-fetch # Conflicts: # docs-mslearn/toolkit/changelog.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary\n- adds the custom - AI billing ADX page into the versioned FinOps Hub dashboard source\n- keeps the existing - Usage optimization (Advisor recommendations) view in place\n- wires AI visuals to shared cost base queries and applies the global billing currency filter\n- normalizes AI KQL service predicates with explicit grouping\n\n## Scope\n- src/templates/finops-hub/dashboard.json\n\n## Notes\n- the AI page is inserted after - Usage optimization under OPTIMIZE\n- includes 7 tiles and 4 AI-specific queries