feat: add Tracing Insights API and FAOS optimization skill references#2155
feat: add Tracing Insights API and FAOS optimization skill references#2155imatiach-msft wants to merge 2 commits into
Conversation
imatiach-msft
requested review from
XOEEst,
XiaofuHuang,
anchenyi,
ankitbko,
jongio and
tendau
as code owners
There was a problem hiding this comment.
Pull request overview
Adds new Microsoft Foundry skill references to support an automated “insights → optimize” loop: detect quality regressions via a Tracing Insights API, then run FAOS optimization, and verify improvement. This extends the existing microsoft-foundry skill’s trace and observe workflows with new entry points and reference docs.
Changes:
- Expanded
microsoft-foundryskill routing keywords and updated sub-skill summaries to include FAOS + tracing insights scenarios. - Added new entry points in
trace.mdandobserve.mdto route users directly to the new references. - Introduced three new reference docs: Tracing Insights API, FAOS optimization, and an end-to-end insights-to-optimize workflow.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| plugin/skills/microsoft-foundry/SKILL.md | Updates description and sub-skill table entries to include FAOS + tracing insights routing. |
| plugin/skills/microsoft-foundry/foundry-agent/trace/trace.md | Adds a table entry routing “auto-detect agent issues” to the Tracing Insights API reference. |
| plugin/skills/microsoft-foundry/foundry-agent/trace/references/tracing-insights-api.md | New API reference for automated changepoint/anomaly detection insights from App Insights eval data. |
| plugin/skills/microsoft-foundry/foundry-agent/observe/observe.md | Adds table entries routing to FAOS optimization and the insights-to-optimize loop references. |
| plugin/skills/microsoft-foundry/foundry-agent/observe/references/faos-optimization.md | New reference documenting FAOS endpoint, request/response shape, and polling. |
| plugin/skills/microsoft-foundry/foundry-agent/observe/references/insights-to-optimize.md | New orchestration reference tying Tracing Insights → FAOS → verification together. |
jongio
left a comment
jongio
left a comment
There was a problem hiding this comment.
CI is blocking on two issues:
-
SKILL.md description is 1190 chars (max 1024). The new FAOS/insights keywords pushed it over. Trim or split some trigger phrases into sub-skill routing instead of frontmatter.
-
Markdown References check fails due to the broken relative path in tracing-insights-api.md (already noted by the bot - fix is
../../observe/references/not../../../observe/references/).
One additional note on the new API endpoints below.
imatiach-msft
force-pushed
the
feat/tracing-insights-faos-skills
branch
5 times, most recently
from
fccf1ec to
9fa20bd
Compare
Add skill reference files for the automated insights-to-optimize loop: - tracing-insights-api.md: Tracing Insights API for anomaly detection - faos-optimization.md: FAOS prompt optimization service - insights-to-optimize.md: End-to-end insights-to-optimize workflow Update entry points in trace.md, observe.md, and SKILL.md with new routing keywords (FAOS, tracing insights, quality regression, anomaly detection). Add preview endpoints callout. Use placeholders instead of hardcoded subscription values. Update trigger test snapshots. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
imatiach-msft
force-pushed
the
feat/tracing-insights-faos-skills
branch
from
9fa20bd to
5c21283
Compare
jongio
left a comment
jongio
left a comment
There was a problem hiding this comment.
Addresses my previous feedback. Description trimmed within the 1024-char limit, preview callouts added for both EUAP endpoints, and the other fixes (relative paths, ARM format, $opId capture, placeholder substitution) all look correct. CI is green. Clean docs.
- tracing-insights-api.md: Update endpoint from v1-beta1 to v1-beta2, document new agents[] response structure, add relatedSpans with operationId for trace querying, add KQL snippet for fetching traces - insights-to-optimize.md: Add Step 2 for extracting traces from relatedSpans operationIds before building FAOS dataset Validated E2E on ilmat-2951: 11 insights, 49 spans with operationIds, 9/12 traces with full content from App Insights. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jongio
left a comment
jongio
left a comment
There was a problem hiding this comment.
The v1-beta2 update looks correct. The new relatedSpans with operationId for trace correlation is a solid addition, and the KQL snippets for extracting traces from App Insights are well-structured. The new Step 2 in the insights-to-optimize flow logically slots in between insight detection and FAOS dataset construction, with a sensible fallback for empty relatedSpans.
All my earlier feedback (description length, preview callouts, relative paths, ARM format, $opId capture, placeholder substitution) remains addressed. CI is green.
jongio
left a comment
jongio
left a comment
There was a problem hiding this comment.
All prior feedback addressed. v1-beta2 API update with relatedSpans is clean and E2E validated. CI green.
|
@jongio it looks like I don't have permissions to merge the PR, how can I get those permissions? or what is the process to merge? |
|
@jongio @XOEEst gentle ping on this PR. I have sent traces to @XOEEst in teams as well showing an e2e scenario where the skill files work. Note that FAOS is moving to a project endpoint but I can move to that new API once it's ready, that should be a relatively minor change on top of the current PR's set of changes. |
Summary
Adds production skill files for the automated insights-to-optimize loop (Tracing Insights API to FAOS to improved agent). This moves the validated POC from prompt-optimization/experiments/TracingInsightsE2EPOC into the production skill tree.
Changes
New reference files:
Updated files:
Scope
Prompt agents only. Hosted agent support is deferred per FAOS Build scope decisions (see Agent insights syncup).
Token limits
All new reference files are within the 1000-token limit for references/**/*.md:
Background