Skip to content
This repository was archived by the owner on Mar 25, 2026. It is now read-only.

Commit 7ee5cdd

Browse files
afterrburnafterrburn
andauthored
QA Agent Interface (#208)
* Remove unnecessary validate files system * move config to agent-dcs dir * remove config from root dir * Add prompt classifier * enhance RAG prompt * Add simple search dialog * remove prompt type parsing * properly rendering Markdown and retrieved documents * Facelift for AI search dialog * Coderabbit suggested cleanups * fix breaking build * move config to env variable * small clean up * abort controller on component dismount * break custom search dialog into multiple modules * linter error fix * Result format validation * simplify key down handler * enhancing the doc navigation with header locator * update readme and .env.example * update readme * outside contributor instruction --------- Co-authored-by: afterrburn <sun_rsh@outlook.com>
1 parent 14556de commit 7ee5cdd

26 files changed

Lines changed: 1576 additions & 200 deletions

.env.example

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Agent Configuration
2+
AGENT_BASE_URL=http://127.0.0.1:3500
3+
AGENT_ID=agent_9ccc5545e93644bd9d7954e632a55a61
4+
5+
# Alternative: You can also set the full URL instead of BASE_URL + ID
6+
# AGENT_FULL_URL=http://127.0.0.1:3500/agent_9ccc5545e93644bd9d7954e632a55a61
7+
8+
# Next.js Environment
9+
NEXTJS_ENV=development

.github/workflows/sync-docs-full.yml

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,7 @@ jobs:
1212
- name: Collect and validate files
1313
run: |
1414
set -euo pipefail
15-
./bin/collect-all-files.sh | \
16-
./bin/validate-files.sh > all-files.txt
15+
./bin/collect-all-files.sh > all-files.txt
1716
1817
echo "Files to sync:"
1918
cat all-files.txt

.github/workflows/sync-docs.yml

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,7 @@ jobs:
1919
run: |
2020
set -euo pipefail
2121
git fetch origin ${{ github.event.before }}
22-
./bin/collect-changed-files.sh "${{ github.event.before }}" "${{ github.sha }}" | \
23-
./bin/validate-files.sh > changed-files.txt
22+
./bin/collect-changed-files.sh "${{ github.event.before }}" "${{ github.sha }}" > changed-files.txt
2423
2524
echo "Files to sync:"
2625
cat changed-files.txt

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,8 @@ yarn-error.log*
2424

2525
# others
2626
.env*.local
27+
.env.local
28+
.env.production
2729
.vercel
2830
next-env.d.ts
2931
.open-next

README.md

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,30 @@
88

99
This project contains the Agentuity documentation website, created using Fumadocs and running on NextJS 15.
1010

11-
## Running
11+
To make the search feature work, you must set up `.env.local` with the following steps.
12+
13+
## Quick Start Guide
14+
15+
1. **Navigate to the Agent Directory:**
16+
```bash
17+
cd agent-docs
18+
```
19+
20+
2. **Start the Agent:**
21+
```bash
22+
agentuity dev
23+
```
24+
25+
3. **Copy Environment Configuration:**
26+
For local development, copy the `.env.example` file to `.env.local`:
27+
```bash
28+
cp .env.example .env.local
29+
```
30+
31+
4. **Update `AGENT_ID`:**
32+
If you are a contributor from outside the Agentuity organization, ensure that you update the `AGENT_ID` in your `.env.local` file with your specific agent ID from the `agentuity dev` run.
33+
34+
## Running Docs Application
1235

1336
```bash
1437
npm run dev
File renamed without changes.

agent-docs/src/agents/doc-processing/docs-orchestrator.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import type { AgentContext } from '@agentuity/sdk';
22
import { processDoc } from './docs-processor';
3-
import { VECTOR_STORE_NAME } from '../../../../config';
3+
import { VECTOR_STORE_NAME } from '../../../config';
44
import type { SyncPayload, SyncStats } from './types';
55

66
/**
Lines changed: 15 additions & 108 deletions
Original file line numberDiff line numberDiff line change
@@ -1,122 +1,29 @@
11
import type { AgentContext, AgentRequest, AgentResponse } from '@agentuity/sdk';
2-
import { streamText } from 'ai';
3-
import { openai } from '@ai-sdk/openai';
4-
5-
import type { ChunkMetadata } from '../doc-processing/types';
6-
import { VECTOR_STORE_NAME, vectorSearchNumber } from '../../../../config';
7-
import type { RelevantDoc } from './types';
2+
import answerQuestion from './rag';
83

94
export default async function Agent(
105
req: AgentRequest,
116
resp: AgentResponse,
127
ctx: AgentContext
138
) {
14-
const prompt = await req.data.text();
15-
const relevantDocs = await retrieveRelevantDocs(ctx, prompt);
16-
17-
const systemPrompt = `
18-
You are a developer documentation assistant. Your job is to answer user questions about the Agentuity platform as effectively and concisely as possible, adapting your style to the user's request. If the user asks for a direct answer, provide it without extra explanation. If they want an explanation, provide a clear and concise one. Use only the provided relevant documents to answer.
19-
20-
You must not make up answers if the provided documents don't exist. You can be direct to the user that the documentations
21-
don't seem to include what they are looking for. Lying to the user is prohibited as it only slows them down. Feel free to
22-
suggest follow up questions if what they're asking for don't seem to have an answer in the document. You can provide them
23-
a few related things that the documents contain that may interest them.
24-
25-
For every answer, return a valid JSON object with:
26-
1. "answer": your answer to the user's question.
27-
2. "documents": an array of strings, representing the path of the documents you used to answer.
28-
29-
If you use information from a document, include it in the "documents" array. If you do not use any documents, return an empty array for "documents".
30-
31-
User question:
32-
\`\`\`
33-
${prompt}
34-
\`\`\`
9+
let jsonRequest: any = null;
10+
let prompt: string;
3511

36-
Relevant documents:
37-
${JSON.stringify(relevantDocs, null, 2)}
38-
39-
Respond ONLY with a valid JSON object as described above. In your answer, you should format code blocks properly in Markdown style if the user needs answer in code block.
40-
`.trim();
41-
42-
const llmResponse = await streamText({
43-
model: openai('gpt-4o'),
44-
system: systemPrompt,
45-
prompt: prompt,
46-
maxTokens: 2048,
47-
});
48-
49-
return resp.stream(llmResponse.textStream);
50-
}
51-
52-
async function retrieveRelevantDocs(ctx: AgentContext, prompt: string): Promise<RelevantDoc[]> {
53-
const dbQuery = {
54-
query: prompt,
55-
limit: vectorSearchNumber
56-
}
5712
try {
58-
59-
60-
const vectors = await ctx.vector.search(VECTOR_STORE_NAME, dbQuery);
61-
62-
const uniquePaths = new Set<string>();
63-
64-
vectors.forEach(vec => {
65-
if (!vec.metadata) {
66-
ctx.logger.warn('Vector missing metadata');
67-
return;
68-
}
69-
const path = typeof vec.metadata.path === 'string' ? vec.metadata.path : undefined;
70-
if (!path) {
71-
ctx.logger.warn('Vector metadata path is not a string');
72-
return;
73-
}
74-
uniquePaths.add(path);
75-
});
76-
77-
const docs = await Promise.all(
78-
Array.from(uniquePaths).map(async path => ({
79-
path,
80-
content: await retrieveDocumentBasedOnPath(ctx, path)
81-
}))
82-
);
83-
84-
return docs;
85-
} catch (err) {
86-
ctx.logger.error('Error retrieving relevant docs: %o', err);
87-
return [];
88-
}
89-
}
90-
91-
async function retrieveDocumentBasedOnPath(ctx: AgentContext, path: string): Promise<string> {
92-
const dbQuery = {
93-
query: ' ',
94-
limit: 10000,
95-
metadata: {
96-
path: path
13+
jsonRequest = await req.data.json();
14+
if (typeof jsonRequest === 'object' && jsonRequest !== null && 'message' in jsonRequest) {
15+
prompt = String(jsonRequest.message || '');
16+
} else {
17+
prompt = JSON.stringify(jsonRequest);
9718
}
19+
} catch {
20+
prompt = await req.data.text();
9821
}
99-
try {
100-
const vectors = await ctx.vector.search(VECTOR_STORE_NAME, dbQuery);
101-
102-
// Sort vectors by chunk index and concatenate text
103-
const sortedVectors = vectors
104-
.map(vec => {
105-
const metadata = vec.metadata as ChunkMetadata;
106-
return {
107-
metadata,
108-
index: metadata.chunkIndex
109-
};
110-
})
111-
.sort((a, b) => a.index - b.index);
11222

113-
const fullText = sortedVectors
114-
.map(vec => vec.metadata.text)
115-
.join('\n\n');
116-
117-
return fullText;
118-
} catch (err) {
119-
ctx.logger.error('Error retrieving document by path %s: %o', path, err);
120-
return '';
23+
if (!prompt.trim()) {
24+
return resp.text("How can I help you?");
12125
}
26+
27+
const answer = await answerQuestion(ctx, prompt);
28+
return resp.json(answer);
12229
}
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
import type { AgentContext } from '@agentuity/sdk';
2+
import { generateObject } from 'ai';
3+
import { openai } from '@ai-sdk/openai';
4+
import type { PromptType } from './types';
5+
import { PromptClassificationSchema } from './types';
6+
7+
/**
8+
* Determines the prompt type based on the input string using LLM classification.
9+
* Uses specific, measurable criteria to decide between Normal and Agentic RAG.
10+
* @param ctx - Agent Context for logging and LLM access
11+
* @param input - The input string to analyze
12+
* @returns {Promise<PromptType>} - The determined PromptType
13+
*/
14+
export async function getPromptType(ctx: AgentContext, input: string): Promise<PromptType> {
15+
const systemPrompt = `
16+
You are a query classifier that determines whether a user question requires simple retrieval (Normal) or complex reasoning (Thinking).
17+
18+
Use these SPECIFIC criteria for classification:
19+
20+
**THINKING (Agentic RAG) indicators:**
21+
- Multi-step reasoning required (e.g., "compare and contrast", "analyze pros/cons")
22+
- Synthesis across multiple concepts (e.g., "how does X relate to Y")
23+
- Scenario analysis (e.g., "what would happen if...", "when should I use...")
24+
- Troubleshooting/debugging questions requiring logical deduction
25+
- Questions with explicit reasoning requests ("explain why", "walk me through")
26+
- Comparative analysis ("which is better for...", "what are the trade-offs")
27+
28+
**NORMAL (Simple RAG) indicators:**
29+
- Direct factual lookups (e.g., "what is...", "how do I install...")
30+
- Simple how-to questions with clear answers
31+
- API reference queries
32+
- Configuration/syntax questions
33+
- Single-concept definitions
34+
35+
Respond with a JSON object containing:
36+
- type: "Normal" or "Thinking"
37+
- confidence: 0.0-1.0 (how certain you are)
38+
- reasoning: brief explanation of your classification
39+
40+
Be conservative - when in doubt, default to "Normal" for better performance.`;
41+
42+
try {
43+
const result = await generateObject({
44+
model: openai('gpt-4o-mini'), // Use faster model for classification
45+
system: systemPrompt,
46+
prompt: `Classify this user query: "${input}"`,
47+
schema: PromptClassificationSchema,
48+
maxTokens: 200,
49+
});
50+
51+
ctx.logger.info('Prompt classified as %s (confidence: %f): %s',
52+
result.object.type, result.object.confidence, result.object.reasoning);
53+
54+
return result.object.type as PromptType;
55+
56+
} catch (error) {
57+
ctx.logger.error('Error classifying prompt, defaulting to Normal: %o', error);
58+
return 'Normal' as PromptType;
59+
}
60+
}
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
import type { AgentContext } from '@agentuity/sdk';
2+
import { generateObject } from 'ai';
3+
import { openai } from '@ai-sdk/openai';
4+
5+
import { retrieveRelevantDocs } from './retriever';
6+
import { AnswerSchema } from './types';
7+
import type { Answer } from './types';
8+
9+
export default async function answerQuestion(ctx: AgentContext, prompt: string) {
10+
const relevantDocs = await retrieveRelevantDocs(ctx, prompt);
11+
12+
const systemPrompt = `
13+
You are Agentuity's developer-documentation assistant.
14+
15+
=== RULES ===
16+
1. Use ONLY the content inside <DOCS> tags to craft your reply. If the required information is missing, state that the docs do not cover it.
17+
2. Never fabricate or guess undocumented details.
18+
3. Ambiguity handling:
19+
• When <DOCS> contains more than one distinct workflow or context that could satisfy the question, do **not** choose for the user.
20+
• Briefly (≤ 2 sentences each) summarise each plausible interpretation and ask **one** clarifying question so the user can pick a path.
21+
• Provide a definitive answer only after the ambiguity is resolved.
22+
4. Answer style:
23+
• If the question can be answered unambiguously from a single workflow, give a short, direct answer.
24+
• Add an explanation only when the user explicitly asks for one.
25+
• Format your response in **MDX (Markdown Extended)** format with proper syntax highlighting for code blocks.
26+
• Use appropriate headings (##, ###) to structure longer responses.
27+
• Wrap CLI commands in \`\`\`bash code blocks for proper syntax highlighting.
28+
• Wrap code snippets in appropriate language blocks (e.g., \`\`\`typescript, \`\`\`json, \`\`\`javascript).
29+
• Use **bold** for important terms and *italic* for emphasis when appropriate.
30+
• Use > blockquotes for important notes or warnings.
31+
5. You may suggest concise follow-up questions or related topics that are present in <DOCS>.
32+
6. Keep a neutral, factual tone.
33+
34+
=== OUTPUT FORMAT ===
35+
Return **valid JSON only** matching this TypeScript type:
36+
37+
type LlmAnswer = {
38+
answer: string; // The reply in MDX format or the clarifying question
39+
documents: string[]; // Paths of documents actually cited
40+
}
41+
42+
The "answer" field should contain properly formatted MDX content that will render beautifully in a documentation site.
43+
The "documents" field must contain the path to the documents you used to answer the question. On top of the path, you may include a specific heading of the document so that the navigation will take the user to the exact point of the document you reference. To format the heading, use the following convention: append the heading to the path using a hash symbol (#) followed by the heading text, replacing spaces with hyphens (-) and converting all characters to lowercase. If there are multiple identical headings, append an index to the heading in the format -index (e.g., #example-3 for the third occurrence of "Example"). For example, if the document path is "/docs/guide" and the heading is "Getting Started", the formatted path would be "/docs/guide#getting-started".
44+
If you cited no documents, return an empty array. Do NOT wrap the JSON in Markdown or add any extra keys.
45+
46+
=== MDX FORMATTING EXAMPLES ===
47+
For CLI commands:
48+
\`\`\`bash
49+
agentuity agent create my-agent "My agent description" bearer
50+
\`\`\`
51+
52+
For code examples:
53+
\`\`\`typescript
54+
import type { AgentRequest, AgentResponse, AgentContext } from "@agentuity/sdk";
55+
56+
export default async function Agent(req: AgentRequest, resp: AgentResponse, ctx: AgentContext) {
57+
return resp.json({hello: 'world'});
58+
}
59+
\`\`\`
60+
61+
For structured responses:
62+
## Creating a New Agent
63+
64+
To create a new agent, use the CLI command:
65+
66+
\`\`\`bash
67+
agentuity agent create [name] [description] [auth_type]
68+
\`\`\`
69+
70+
**Parameters:**
71+
- \`name\`: The agent name
72+
- \`description\`: Agent description
73+
- \`auth_type\`: Either \`bearer\` or \`none\`
74+
75+
> **Note**: This command will create the agent in the Agentuity Cloud and set up local files.
76+
77+
<QUESTION>
78+
${prompt}
79+
</QUESTION>
80+
81+
<DOCS>
82+
${JSON.stringify(relevantDocs, null, 2)}
83+
</DOCS>
84+
`;
85+
86+
try {
87+
const result = await generateObject({
88+
model: openai('gpt-4o'),
89+
system: systemPrompt,
90+
prompt: prompt,
91+
schema: AnswerSchema,
92+
maxTokens: 2048,
93+
});
94+
return result.object;
95+
} catch (error) {
96+
ctx.logger.error('Error generating answer: %o', error);
97+
98+
// Fallback response with MDX formatting
99+
const fallbackAnswer: Answer = {
100+
answer: `## Error
101+
102+
I apologize, but I encountered an error while processing your question.
103+
104+
**Please try:**
105+
- Rephrasing your question
106+
- Being more specific about what you're looking for
107+
- Checking if your question relates to Agentuity's documented features
108+
109+
> If the problem persists, please contact support.`,
110+
documents: []
111+
};
112+
113+
return fallbackAnswer;
114+
}
115+
}

0 commit comments

Comments
 (0)