Skip to content

Fix: Prices transform cross-row operations broken by per-batch execution #2037

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
RolandKrummenacher wants to merge 5 commits into from
Closed

Fix: Prices transform cross-row operations broken by per-batch execution #2037

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2d117d6
Fix: Prices transform cross-row operations broken by per-batch execut…
Mar 2, 2026
8088d64
chore: Add changelog entry for prices transform fix (#1625)
Mar 2, 2026
807ac12
chore: Update ms.date in docs-mslearn files
github-actions[bot] Mar 2, 2026
e036c2c
Fix v1_0 comment references and simplify changelog wording
Mar 15, 2026
3aa2856
Move ContractedUnitPrice/ListUnitPrice into project-rename block
Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docs-mslearn/toolkit/changelog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: FinOps toolkit changelog
description: Review the latest features and enhancements in the FinOps toolkit, including updates to FinOps hubs, Power BI reports, and more.
author: MSBrett
ms.author: brettwil
ms.date: 03/01/2026
ms.date: 03/02/2026
ms.topic: reference
ms.service: finops
ms.subservice: finops-toolkit
Expand Down Expand Up @@ -46,6 +46,10 @@ The following section lists features and enhancements that are currently in deve
- Improved deployment UI to consolidate hub mode selection into a single radio button group with four mutually exclusive options: None (storage only for Power BI reports), Azure Data Explorer, Microsoft Fabric, or Remote Hub ([#1929](https://github.com/microsoft/finops-toolkit/issues/1929)).
- Remote Hub configuration (storage URI, storage key, and purge protection) is now displayed in the Basics tab when Remote Hub mode is selected, making the mutual exclusivity clear.
- Data Explorer SKU and retention settings are now only visible when Azure Data Explorer mode is selected.
- **Fixed**
- Fixed price data ingestion bugs when price sheet data is split across multiple files ([#1625](https://github.com/microsoft/finops-toolkit/issues/1625)).
- Commitment discount prices are now consistently set correctly using a full price sheet lookup.
- Commitment discount eligibility columns moved to the Prices* Hub functions rather than hard-coded in the `Prices_final_v*` tables.

### [PowerShell module](powershell/powershell-commands.md) v14

Expand Down
17 changes: 16 additions & 1 deletion src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/HubSetup_v1_0.kql
View file
Open in desktop
Copy link
Copy Markdown
Collaborator

@flanakin flanakin Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fwiw, I'm slightly torn on this... In one sense, I like providing this and I believe we'll have to do it for FOCUS 1.4 anyway, but I also hesitate to add this to EVERY Prices call. Given it's not working today, I would argue we could leave it empty and plan a separate fix based on priority. We just need to create a bug to track it.

I'm curious what others think.

Copy link
Copy Markdown
Collaborator Author

@RolandKrummenacher RolandKrummenacher Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — let's defer this. Will remove the eligibility computation from the Hub views and create a separate issue to track it.

Original file line number Diff line number Diff line change
Expand Up @@ -216,10 +216,22 @@ Costs_v1_0()


// Prices_final_v1_0
// Commitment discount eligibility is computed here instead of in the update policy because ADX
// update policies execute per-ingestion batch, not across the full table.
// See https://github.com/microsoft/finops-toolkit/issues/1625
.create-or-alter function
with (docstring = 'Gets all prices aligned to FOCUS 1.0.', folder = 'Prices')
Prices_v1_0()
{
// Commitment discount eligibility -- computed here because the update policy only sees per-batch data
let commitmentMeters = database('Ingestion').Prices_final_v1_0
| union (database('Ingestion').Prices_final_v1_2)
| where x_SkuPriceType in ('ReservedInstance', 'SavingsPlan')
| distinct x_SkuMeterId, x_SkuPriceType;
let riMeters = commitmentMeters | where x_SkuPriceType == 'ReservedInstance' | project x_SkuMeterId;
let spMeters = commitmentMeters | where x_SkuPriceType == 'SavingsPlan' | project x_SkuMeterId;
//
// Calculate commitment discount eligibility
database('Ingestion').Prices_final_v1_0
| union (
database('Ingestion').Prices_final_v1_2
Expand All @@ -238,12 +250,15 @@ Prices_v1_0()
x_SkuIncludedQuantity = todecimal(x_SkuIncludedQuantity),
x_SkuTier = todecimal(x_SkuTier),
x_TotalUnitPriceDiscount = todecimal(x_TotalUnitPriceDiscount),
x_TotalUnitPriceDiscountPercent = todecimal(x_TotalUnitPriceDiscountPercent)
x_TotalUnitPriceDiscountPercent = todecimal(x_TotalUnitPriceDiscountPercent)
// Rename columns
| project-rename
x_PricingCurrency = PricingCurrency,
x_SkuMeterName = SkuMeter
)
| extend x_CommitmentDiscountSpendEligibility = iff(x_SkuMeterId in (riMeters) and x_SkuPriceType == 'Consumption', 'Eligible', 'Not Eligible')
| extend x_CommitmentDiscountUsageEligibility = iff(x_SkuMeterId in (spMeters) and x_SkuPriceType == 'Consumption', 'Eligible', 'Not Eligible')
//
| project
BillingAccountId,
BillingAccountName,
Expand Down
17 changes: 16 additions & 1 deletion src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/HubSetup_v1_2.kql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -360,10 +360,22 @@ Costs_v1_2()


// Prices_final_v1_2
// Commitment discount eligibility is computed here instead of in the update policy because ADX
// update policies execute per-ingestion batch, not across the full table.
// See https://github.com/microsoft/finops-toolkit/issues/1625
.create-or-alter function
with (docstring = 'Gets all prices aligned to FOCUS 1.2.', folder = 'Prices')
Prices_v1_2()
{
// Commitment discount eligibility -- computed here because the update policy only sees per-batch data
let commitmentMeters = database('Ingestion').Prices_final_v1_2
| union (database('Ingestion').Prices_final_v1_0)
| where x_SkuPriceType in ('ReservedInstance', 'SavingsPlan')
| distinct x_SkuMeterId, x_SkuPriceType;
let riMeters = commitmentMeters | where x_SkuPriceType == 'ReservedInstance' | project x_SkuMeterId;
let spMeters = commitmentMeters | where x_SkuPriceType == 'SavingsPlan' | project x_SkuMeterId;
//
// Calculate commitment discount eligibility
database('Ingestion').Prices_final_v1_2
| union (
database('Ingestion').Prices_final_v1_0
Expand All @@ -381,12 +393,15 @@ Prices_v1_2()
x_SkuIncludedQuantity = toreal(x_SkuIncludedQuantity),
x_SkuTier = toreal(x_SkuTier),
x_TotalUnitPriceDiscount = toreal(x_TotalUnitPriceDiscount),
x_TotalUnitPriceDiscountPercent = toreal(x_TotalUnitPriceDiscountPercent)
x_TotalUnitPriceDiscountPercent = toreal(x_TotalUnitPriceDiscountPercent)
// Rename columns
| project-rename
PricingCurrency = x_PricingCurrency,
SkuMeter = x_SkuMeterName
)
| extend x_CommitmentDiscountSpendEligibility = iff(x_SkuMeterId in (riMeters) and x_SkuPriceType == 'Consumption', 'Eligible', 'Not Eligible')
| extend x_CommitmentDiscountUsageEligibility = iff(x_SkuMeterId in (spMeters) and x_SkuPriceType == 'Consumption', 'Eligible', 'Not Eligible')
//
| project
BillingAccountId,
BillingAccountName,
Expand Down
31 changes: 9 additions & 22 deletions ...mplates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_0.kql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ Prices_transform_v1_0()
| extend x_SkuProductId = coalesce(ProductId, ProductID)
| extend x_SkuTerm = isoMonths(Term)
| project-rename
ContractedUnitPrice = UnitPrice,
ListUnitPrice = MarketPrice,
x_BaseUnitPrice = BasePrice,
x_EffectivePeriodEnd = EffectiveEndDate,
x_EffectivePeriodStart = EffectiveStartDate,
Expand All @@ -61,8 +63,6 @@ Prices_transform_v1_0()
x_SkuRegion = MeterRegion,
x_SkuServiceFamily = ServiceFamily,
x_SkuTier = TierMinimumUnits
| extend ContractedUnitPrice = iff(x_SkuPriceType != 'SavingsPlan', UnitPrice, todecimal('')) // UnitPrice for savings plan is not the on-demand unit price
| extend ListUnitPrice = iff(x_SkuPriceType != 'SavingsPlan', MarketPrice, todecimal('')) // MarketPrice for savings plan is not the list price
| extend ChargeCategory = case(
x_SkuPriceType == 'Consumption', 'Usage',
x_SkuPriceType == 'ReservedInstance', 'Purchase',
Expand All @@ -72,38 +72,25 @@ Prices_transform_v1_0()
| extend SkuPriceIdv2 = strcat(case(x_SkuPriceType == 'Consumption', 'OD', x_SkuPriceType == 'ReservedInstance', 'RI', x_SkuPriceType == 'SavingsPlan', 'SP', 'XX'), substring(ChargeCategory, 0, 1), x_SkuTerm, '_', x_SkuProductId, '_', x_SkuId, '_', x_SkuMeterType, '_', x_SkuTier, x_SkuOfferId)
| extend x_BillingAccountId = iff(BillingAccountId startswith '/', split(BillingAccountId, '/')[-1], coalesce(BillingAccountId, EnrollmentNumber))
| extend x_BillingProfileId = iff(BillingProfileId startswith '/', split(BillingProfileId, '/')[-1], coalesce(BillingProfileId, EnrollmentNumber))
| extend tmp_SavingsPlanKey = strcat(x_SkuMeterId, x_SkuProductId, x_SkuId, x_SkuTier, x_SkuOfferId)
//
// Get latest ingested row based on the unique ID
| extend x_IngestionTime = ingestion_time()
);
//
// Meters for reservations and savings plans to identify commitment eligibility
let riMeters = prices | where x_SkuPriceType == 'ReservedInstance' | distinct x_SkuMeterId;
let spMeters = prices | where x_SkuPriceType == 'SavingsPlan' | distinct x_SkuMeterId;
//
// Copy list/base/contracted prices from on-demand SKUs
// NOTE: Commitment discount eligibility requires cross-row lookups and is deferred to the Hub
// view (Prices_v1_0) because ADX update policies execute per-ingestion batch, not across the
// full table. See https://github.com/microsoft/finops-toolkit/issues/1625
Comment thread
RolandKrummenacher marked this conversation as resolved.
prices
| where x_SkuPriceType == 'SavingsPlan'
// If we use join, specify the shuffle key
// TODO: Compare join vs. lookup perf -- | join kind=leftouter hint.strategy=shuffle (prices | where x_SkuPriceType == 'Consumption' | where x_SkuMeterId in (spMeters) | distinct tmp_SavingsPlanKey, ListUnitPrice, ContractedUnitPrice, x_BaseUnitPrice) on tmp_SavingsPlanKey
| lookup kind=leftouter (prices | where x_SkuPriceType == 'Consumption' | where x_SkuMeterId in (spMeters) | distinct tmp_SavingsPlanKey, ListUnitPrice, ContractedUnitPrice, x_BaseUnitPrice) on tmp_SavingsPlanKey
| extend ListUnitPrice = coalesce(ListUnitPrice, ListUnitPrice1)
| extend ContractedUnitPrice = coalesce(ContractedUnitPrice, ContractedUnitPrice1)
| extend x_BaseUnitPrice = coalesce(x_BaseUnitPrice, x_BaseUnitPrice1)
| project-away ListUnitPrice1, ContractedUnitPrice1, x_BaseUnitPrice1, tmp_SavingsPlanKey
| union ((prices | where x_SkuPriceType != 'SavingsPlan'))
//
// Calculate commitment discount elgibility
// TODO: Would a join be faster?
| extend x_CommitmentDiscountSpendEligibility = iff(x_SkuMeterId in (riMeters) and x_SkuPriceType != 'ReservedInstance', 'Eligible', 'Not Eligible')
| extend x_CommitmentDiscountUsageEligibility = iff(x_SkuMeterId in (spMeters), 'Eligible', 'Not Eligible')
// Commitment discount eligibility is computed in the Hub view (Prices_v1_0)
| extend x_CommitmentDiscountSpendEligibility = ''
| extend x_CommitmentDiscountUsageEligibility = ''
//
// Add PricingUnit and x_PricingBlockSize
// TODO: Compare join vs. lookup perf -- | join kind=leftouter (PricingUnits) on x_PricingUnitDescription | project-away x_PricingUnitDescription1
| lookup kind=leftouter (PricingUnits | extend x_PricingBlockSize = todecimal(x_PricingBlockSize)) on x_PricingUnitDescription
//
| extend x_EffectiveUnitPrice = iff(x_SkuPriceType == 'SavingsPlan', UnitPrice, todecimal('')) // Savings plan prices are for the effective price, not the contracted price
| extend x_EffectiveUnitPrice = iff(x_SkuPriceType == 'SavingsPlan', ContractedUnitPrice, todecimal('')) // Savings plan prices are for the effective price, not the contracted price
| extend x_EffectiveUnitPriceDiscount = ContractedUnitPrice - x_EffectiveUnitPrice
| extend x_ContractedUnitPriceDiscount = ListUnitPrice - ContractedUnitPrice
| extend x_TotalUnitPriceDiscount = ListUnitPrice - x_EffectiveUnitPrice
Expand Down
34 changes: 11 additions & 23 deletions ...mplates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_2.kql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ Prices_transform_v1_2()
| extend x_SkuProductId = coalesce(ProductId, ProductID)
| extend x_SkuTerm = isoMonths(Term)
| project-rename
ContractedUnitPrice = UnitPrice,
ListUnitPrice = MarketPrice,
SkuMeter = MeterName,
x_BaseUnitPrice = BasePrice,
x_EffectivePeriodEnd = EffectiveEndDate,
Expand All @@ -52,8 +54,6 @@ Prices_transform_v1_2()
x_SkuRegion = MeterRegion,
x_SkuServiceFamily = ServiceFamily,
x_SkuTier = TierMinimumUnits
| extend ContractedUnitPrice = iff(x_SkuPriceType != 'SavingsPlan', UnitPrice, real(null)) // UnitPrice for savings plan is not the on-demand unit price
| extend ListUnitPrice = iff(x_SkuPriceType != 'SavingsPlan', MarketPrice, real(null)) // MarketPrice for savings plan is not the list price
| extend ChargeCategory = case(
x_SkuPriceType == 'Consumption', 'Usage',
x_SkuPriceType == 'ReservedInstance', 'Purchase',
Expand All @@ -63,27 +63,15 @@ Prices_transform_v1_2()
| extend SkuPriceIdv2 = strcat(case(x_SkuPriceType == 'Consumption', 'OD', x_SkuPriceType == 'ReservedInstance', 'RI', x_SkuPriceType == 'SavingsPlan', 'SP', 'XX'), substring(ChargeCategory, 0, 1), x_SkuTerm, '_', x_SkuProductId, '_', x_SkuId, '_', x_SkuMeterType, '_', x_SkuTier, x_SkuOfferId)
| extend x_BillingAccountId = iff(BillingAccountId startswith '/', split(BillingAccountId, '/')[-1], coalesce(BillingAccountId, EnrollmentNumber))
| extend x_BillingProfileId = iff(BillingProfileId startswith '/', split(BillingProfileId, '/')[-1], coalesce(BillingProfileId, EnrollmentNumber))
| extend tmp_SavingsPlanKey = strcat(x_SkuMeterId, x_SkuProductId, x_SkuId, x_SkuTier, x_SkuOfferId)
//
// Get latest ingested row based on the unique ID
| extend x_IngestionTime = ingestion_time()
);
//
// Meters for reservations and savings plans to identify commitment eligibility
let riMeters = prices | where x_SkuPriceType == 'ReservedInstance' | distinct x_SkuMeterId;
let spMeters = prices | where x_SkuPriceType == 'SavingsPlan' | distinct x_SkuMeterId;
//
// Copy list/base/contracted prices from on-demand SKUs
// NOTE: Commitment discount eligibility requires cross-row lookups and is deferred to the Hub
// view (Prices_v1_2) because ADX update policies execute per-ingestion batch, not across the
// full table. See https://github.com/microsoft/finops-toolkit/issues/1625
prices
| where x_SkuPriceType == 'SavingsPlan'
// If we use join, specify the shuffle key
// TODO: Compare join vs. lookup perf -- | join kind=leftouter hint.strategy=shuffle (prices | where x_SkuPriceType == 'Consumption' | where x_SkuMeterId in (spMeters) | distinct tmp_SavingsPlanKey, ListUnitPrice, ContractedUnitPrice, x_BaseUnitPrice) on tmp_SavingsPlanKey
| lookup kind=leftouter (prices | where x_SkuPriceType == 'Consumption' | where x_SkuMeterId in (spMeters) | distinct tmp_SavingsPlanKey, ListUnitPrice, ContractedUnitPrice, x_BaseUnitPrice) on tmp_SavingsPlanKey
| extend ListUnitPrice = coalesce(ListUnitPrice, ListUnitPrice1)
| extend ContractedUnitPrice = coalesce(ContractedUnitPrice, ContractedUnitPrice1)
| extend x_BaseUnitPrice = coalesce(x_BaseUnitPrice, x_BaseUnitPrice1)
| project-away ListUnitPrice1, ContractedUnitPrice1, x_BaseUnitPrice1, tmp_SavingsPlanKey
| union ((prices | where x_SkuPriceType != 'SavingsPlan'))
//
// Set CommitmentDiscountCategory for reuse
| extend CommitmentDiscountCategory = case(
Expand All @@ -92,11 +80,11 @@ Prices_transform_v1_2()
''
)
//
// Calculate commitment discount eligibility
// TODO: Would a join be faster?
// TODO: Check this to ensure it's correct
| extend x_CommitmentDiscountSpendEligibility = iff(x_SkuMeterId in (riMeters) and x_SkuPriceType != 'ReservedInstance', 'Eligible', 'Not Eligible')
| extend x_CommitmentDiscountUsageEligibility = iff(x_SkuMeterId in (spMeters), 'Eligible', 'Not Eligible')
// Commitment discount eligibility is computed in the Hub view (Prices_v1_2)
// because it requires cross-row lookups across the full price table.
// See https://github.com/microsoft/finops-toolkit/issues/1625
| extend x_CommitmentDiscountSpendEligibility = ''
| extend x_CommitmentDiscountUsageEligibility = ''
//
// TODO: Implement x_CommitmentDiscountNormalizedRatio
| extend x_CommitmentDiscountNormalizedRatio = real(null)
Expand All @@ -105,7 +93,7 @@ Prices_transform_v1_2()
// TODO: Compare join vs. lookup perf -- | join kind=leftouter (PricingUnits) on x_PricingUnitDescription | project-away x_PricingUnitDescription1
| lookup kind=leftouter (PricingUnits) on x_PricingUnitDescription
//
| extend x_EffectiveUnitPrice = iff(x_SkuPriceType == 'SavingsPlan', UnitPrice, real(null)) // Savings plan prices are for the effective price, not the contracted price
| extend x_EffectiveUnitPrice = iff(x_SkuPriceType == 'SavingsPlan', ContractedUnitPrice, real(null)) // Savings plan prices are for the effective price, not the contracted price
| extend x_EffectiveUnitPriceDiscount = ContractedUnitPrice - x_EffectiveUnitPrice
| extend x_ContractedUnitPriceDiscount = ListUnitPrice - ContractedUnitPrice
| extend x_TotalUnitPriceDiscount = ListUnitPrice - x_EffectiveUnitPrice
Expand Down
Loading