Configure Hub to use Log Analytics Workspace for Ingestion vs ADX

[BenTheCloudGuy]

📝 Scenario

Configure Hub to use Log Analytics Workspace for Ingestion vs ADX. This would allow KQL and Dashboards to be used without deploying the required ADX infra. Grafana has a built in datasource for LAW and can leverage KQL to build advanced dashboards and reporting filters that can be shared much easier than PowerBI (and they look better too)

💎 Solution

Work out how we can move FOCUS cost exports to Log Analytics Workspace in custom table via FunctionApp. My thoughts would be to send to storage account as normal, wire up event trigger for when new blob is uploaded to fire our functionApp to feed that data into Log Analytics Workspace. This can easily be wrapped into existing deployment tool and/or IaC.

📋 Tasks

Required tasks

• Review feasibility of this process.

Stretch goals

• Include prebuilt Azure and Azure Managed Grafana Dashboards.

🙋‍♀️ Ask for the communty

We could use your help:

1. Please vote this issue up (👍) to prioritize it.
2. Leave comments to help us solidify the vision.

[microsoft-github-policy-service]

Thank you for submitting this issue! We try to respond within 3 business days. In the meantime:

• 📖 Check the troubleshooting guide for common issues
• 💬 Ask questions in Discussions
• 📅 Join our biweekly office hours for live help from the team

[FallenHoot]

1. Feasibility

• LAW: Suitable for moderate data volumes and short retention (operational analytics, KQL, Data Collector API). Not recommended for large-scale, long-term, or advanced analytics due to ingestion, retention, and query performance limits.
• ADX: Optimized for big data, high-throughput, advanced analytics, and native ADF connector. Scales efficiently for large FOCUS exports and long-term retention.

2. Transformation Options

• Azure Data Factory (ADF): Recommended for robust ETL/ELT, schema mapping, and orchestration. Handles FOCUS complexity and scale.
• Azure Functions: Not suitable for FOCUS due to complexity and scale.

2.5 Proposed Workflow for FOCUS Data Ingestion to Log Analytics Workspace (LAW):

• Export FOCUS dataset to Azure Storage (CSV/Parquet, partitioned).
• ADF pipeline triggers on new blob or scheduled interval.
• ADF Data Flow/Copy Activity reads partitioned files, parses manifest.json.
• Transformation: Use Mapping Data Flows for schema mapping, type conversion, and enrichment.
• Sink: Write to LAW custom table via Azure Monitor Data Collector API or direct integration.
• Validation: Query LAW with KQL to verify ingestion and enable dashboards (Grafana, Azure Monitor).

3. Cost Modeling

• LAW: Storage costs scale with data volume, but ingestion and query performance are limiting factors. Suitable for up to ~250 GB/month and short retention.
• ADX: Costs are ~95% fixed (cluster), not variable with cloud spend. Data ingestion/storage costs are negligible compared to cluster cost.

ADX Data Ingestion & Retention Costs

Monthly Cloud Spend	Data Volume	Ingestion/Month	1-Year ADX Storage	3-Year ADX Storage
$200K	~3 GB/mo	~$0.15	~$0.75	~$2.25
$500K	~8 GB/mo	~$0.40	~$2	~$6
$800K	~14 GB/mo	~$0.70	~$3.50	~$10.50
$1M	~18 GB/mo	~$0.90	~$4.50	~$13.50
$2M	~35 GB/mo	~$1.75	~$8.75	~$26
$5M	~85 GB/mo	~$4.25	~$21	~$64
$10M	~175 GB/mo	~$8.75	~$44	~$131

• Total Cost Example: For $1M/mo cloud spend, FinOps Hub infrastructure is ~$405/mo, with data costs negligible compared to fixed cluster cost.

LAW Data Ingestion & Retention Costs

Monthly Cloud Spend	Est. Data/Month	Storage Cost/Month	Storage Cost/Year
$200K	2-5 GB	$0.10-0.25	$1.20-3
$500K	5-12 GB	$0.25-0.60	$3-7
$800K	8-20 GB	$0.40-1	$5-12
$1M	10-25 GB	$0.50-1.25	$6-15
$2M	20-50 GB	$1-2.50	$12-30
$5M	50-125 GB	$2.50-6.25	$30-75
$10M	100-250 GB	$5-12.50	$60-150

• Total Cost Example: For $1M/mo cloud spend, LAW storage cost is $0.50-1.25/mo, $6-15/year. Ingestion/query costs may be higher for large volumes or long retention.

4. Grafana Integration

• Both LAW and ADX can be used as data sources for Grafana dashboards.
• ADX is preferred for advanced analytics and scalability.

5. Practical Recommendations

• Use LAW for moderate data volumes, short retention, and operational analytics.
• Use ADX for large-scale, long-term, or advanced analytics needs.
• Start with storage-only for dev/test, upgrade to ADX as analytics needs grow.
• Use ADF for transformation; avoid Functions for FOCUS.
• Pause ADX when not in use to optimize costs.

6. When is LAW Not Feasible?

LAW is cost-effective for moderate data volumes (up to ~250 GB/month) and short retention (up to 13 months), but may become impractical for higher volumes or longer retention due to cost and platform limitations. Query performance and ingestion rates are subject to Azure platform limits and pricing. For advanced analytics, high-throughput queries, or long-term retention, ADX is recommended.

[flanakin]

+ @helderpinto who has experience with LAW ingestion as part of AOE
+ @arthurclares who recently requested this as a potential option for simpler demo environments

[helderpinto]

I tend to favor the workflow proposed by @FallenHoot, because it reuses a good part of Hubs ETL, being LAW "just" a variation. Regarding LAW-based reporting, we could build a KQL Power BI variation of the ADX reports (preferable), or build new Azure Monitor workbooks / Grafana dashboards.