Fix: Prices transform cross-row operations broken by per-batch execution

[RolandKrummenacher]

Summary

Fixes #1625 — The Prices_transform_v1_0 and Prices_transform_v1_2 functions don't work as intended because ADX update policies execute per-ingestion batch, not across the full Prices_raw table. When a price sheet is split across multiple parquet files, cross-row operations only see the current file's data.

This PR fixes two issues:

1. Removed the savings plan → consumption price lookup

The update policy previously nulled out ContractedUnitPrice and ListUnitPrice for savings plan rows, then replaced them with on-demand prices from matching Consumption rows via a self-join. This lookup was broken by per-batch execution (SP and Consumption rows could land in different files).

Why we dropped it rather than moving it: Per the EA and MCA price sheet documentation, savings plan rows already have meaningful values:

• UnitPrice → the SP contracted rate (negotiated, including EA/MCA discounts)
• MarketPrice → the SP retail rate (before negotiated discounts)
• BasePrice → the rate at agreement signing

These are the correct prices for the savings plan price type itself. The old lookup was replacing them with on-demand reference prices to enable "SP vs. on-demand" savings analysis — but that conflated different price semantics on the same row. Comparing SP rates against on-demand rates is a cross-row analysis that belongs at query time, not baked into the data.

With this change, each price row now has its own correct prices regardless of price type (Consumption, ReservedInstance, or SavingsPlan), and discount columns show the EA/MCA negotiated discount for that specific price type.

2. Moved commitment discount eligibility to Hub view functions

The riMeters/spMeters distinct lookups (prices | where x_SkuPriceType == 'ReservedInstance' | distinct x_SkuMeterId) require visibility across all price data to determine which meters have RI/SP variants. These have been moved from the update policy to the Prices_v1_2() and Prices_v1_0() Hub view functions, which query the full final tables.

Additional fix: eligibility is now set only on Consumption rows (x_SkuPriceType == 'Consumption'). Previously, spend eligibility excluded RI rows but usage eligibility did not exclude SP rows. Since there are only three price types (Consumption, ReservedInstance, SavingsPlan), eligibility is meaningful only for on-demand rows — RI/SP rows are the commitment discounts themselves.

Impact on downstream consumers

Prices_v1_2() / Prices_v1_0() ADX functions — these are the public-facing functions that users query directly. The semantic change for SP rows:

Column	Before (when lookup worked)	After this PR
ContractedUnitPrice	Consumption contracted rate	SP contracted rate
ListUnitPrice	Consumption retail rate	SP retail rate
x_EffectiveUnitPrice	SP contracted rate	SP contracted rate (unchanged)
x_EffectiveUnitPriceDiscount	Consumption - SP = SP savings	SP - SP = 0
x_ContractedUnitPriceDiscount	Retail - Contracted (negotiated)	SP retail - SP contracted
x_TotalUnitPriceDiscount	Retail - SP = total savings	SP retail - SP contracted

Note: the old behavior was already broken for most users due to the per-batch issue (#1625). The lookup only produced correct results when all price data landed in a single ingestion batch.

Costs transform functions — The Costs_transform_v1_2 and Costs_transform_v1_0 functions look up prices from Prices_final to populate missing ContractedUnitPrice/ListUnitPrice on cost rows. This lookup is not affected because it filters on x_SkuPriceType == 'Consumption', and Consumption rows were never nulled — only SP rows were.

Power BI reports — Not affected:

• Storage-based reports read raw parquet files and do their own column mapping — they never go through the KQL transform.
• KQL-based reports have no Prices data table; the Costs table gets ContractedUnitPrice/ListUnitPrice from cost exports, not price sheets.

ADX real-time dashboard (dashboard.json) — Not directly affected. The dashboard's SKU prices and cost validation tiles query Costs data (CostsByDay, Costs_v1_2), not the Prices functions. The only Prices reference is a data ingestion monitoring query that checks table extents, not price values.

Files changed

• IngestionSetup_v1_2.kql — Remove SP price nulling, cross-row lookup, and eligibility calculation from update policy
• IngestionSetup_v1_0.kql — Same changes (deprecated but kept consistent)
• HubSetup_v1_2.kql — Add eligibility computation with combined meter lookup across both final tables
• HubSetup_v1_0.kql — Same eligibility computation for FOCUS 1.0 consumers

Test plan

• Deploy to a test hub with a price sheet split across multiple parquet files
• Verify Prices_final_v1_2 has non-null ContractedUnitPrice/ListUnitPrice for all price types including SavingsPlan
• Verify Prices_v1_2() returns correct x_CommitmentDiscountSpendEligibility and x_CommitmentDiscountUsageEligibility on Consumption rows
• Verify RI and SP rows show Not Eligible for both eligibility columns
• Verify SP rows: ContractedUnitPrice = SP contracted rate, ListUnitPrice = SP retail rate
• Verify SP rows: x_EffectiveUnitPriceDiscount = 0 (expected, since ContractedUnitPrice and x_EffectiveUnitPrice are now the same value)
• Verify Consumption rows: discount columns unchanged
• Verify Costs transform still populates missing prices correctly (Consumption lookup unaffected)
• Verify ADX dashboard SKU prices tiles render correctly

🤖 Generated with Claude Code

[flanakin]

🤖 [AI][Claude] PR Review

Summary: Well-structured fix that correctly addresses the per-batch execution limitation in ADX update policies. The approach of moving cross-row eligibility lookups to query-time Hub view functions is sound, and the decision to stop nulling SP price columns is well-justified by the documented schema semantics.

⚠️ Should fix (1)

1. Copy-paste comment in IngestionSetup_v1_0.kql references Prices_v1_2 instead of Prices_v1_0 (lines 81, 85)

✅ What's done well

• Thorough PR description with clear before/after semantic table and downstream impact analysis
• Correct architectural fix — moving cross-row operations to query-time Hub views
• Consistent eligibility logic across both v1_0 and v1_2 paths, with both Hub views unioning across both final tables
• Tightening eligibility to x_SkuPriceType == 'Consumption' only is semantically cleaner than the old asymmetric exclusion logic

[flanakin]

🤖 [AI][Claude] ⚠️ Should fix

These comments reference Prices_v1_2 but this is the v1_0 transform file. Should reference Prices_v1_0 (or generically "the Hub view functions") for consistency. Same issue on line 85 below.

Suggestion:

Suggested change

	// NOTE: Commitment discount eligibility requires cross-row lookups and is deferred to the Hub
	// view (Prices_v1_2) because ADX update policies execute per-ingestion batch, not across the
	// full table. See https://github.com/microsoft/finops-toolkit/issues/1625
	// NOTE: Commitment discount eligibility requires cross-row lookups and is deferred to the Hub
	// view (Prices_v1_0) because ADX update policies execute per-ingestion batch, not across the
	// full table. See https://github.com/microsoft/finops-toolkit/issues/1625

[flanakin]

nit: This is a bit verbose.

Suggested change

	- Fixed prices transform function producing incorrect results when price sheet data is split across multiple ingestion batches by moving cross-row operations (commitment discount eligibility) from the ingestion-time update policy to the query-time Hub view functions ([#1625](https://github.com/microsoft/finops-toolkit/issues/1625)).
	- Fixed price data ingestion bugs when price sheet data is split across multiple files ([#1625](https://github.com/microsoft/finops-toolkit/issues/1625)).
	- Commitment discount prices are now consistently set correctly using a full price sheet lookup.
	- Commitment discount eligibility columns moved to the Prices* Hub functions rather than hard-coded in the `Prices_final_v*` tables.

[flanakin]

Did you validate these are correct for savings plan prices? If memory serves, these didn't have valid values. If they do now for EA and MCA, we should be good to remove the extend. Fwiw, I would move these into the project-rename above, if we are removing this (in alpha order).

[flanakin]

Same comment as in the v1_0 file

[flanakin]

Fwiw, I'm slightly torn on this... In one sense, I like providing this and I believe we'll have to do it for FOCUS 1.4 anyway, but I also hesitate to add this to EVERY Prices call. Given it's not working today, I would argue we could leave it empty and plan a separate fix based on priority. We just need to create a bug to track it.

I'm curious what others think.

[microsoft-github-policy-service]

@@RolandKrummenacher: you have some new feedback!

Please review and resolve all comments and I'll let reviewers know by removing the Needs: Attention label. If I miss anything, just reply with #needs-review and I'll update the status.