feat(ai): add multi-provider AI support with enhanced extraction#547
feat(ai): add multi-provider AI support with enhanced extraction#547kelvinkipruto merged 7 commits intomainfrom
Conversation
- Introduce provider catalog and registry to support multiple AI providers (Google, OpenAI, Anthropic, etc.) - Add provider credentials management in settings with environment variable fallbacks - Implement two-pass extraction with chunking and tool-based normalization - Add force re-extract capability for existing documents - Update dependencies to latest AI SDK versions and Zod 4.3.6 - Deprecate legacy Google-only model configuration in favor of new provider system
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eb9861ef8d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@claude review |
|
Claude finished @kelvinkipruto's task in 2m 19s —— View job Code Review
Overall this is a well-structured feature. The provider catalog/registry split is clean, backward compatibility with legacy config is handled gracefully, and the two-pass extraction with chunking is a meaningful improvement. Below are issues ranging from critical to minor. P1 — Bugs / Data Loss1. Data deleted before new extraction succeeds ( When // Existing extractions deleted here...
for (const extraction of existingExtractions) {
await payload.delete({ collection: "ai-extractions", id: extraction.id });
}
// ...but extraction can still fail after this point
const extractedCandidates = await extractPromisesFromChunks({...});Fix: Defer deletion until after the new 2. All-chunk failures produce false negatives (
} catch (error) {
// ...logged as warn...
if (APICallError.isInstance(error) || NoObjectGeneratedError.isInstance(error)) {
continue; // ← silently drops chunk failure
}
throw error;
}Fix: After the chunk loop, check if P2 — Security / Design Concerns3. API keys stored as plaintext in database ( The Consider using a password/secret field type, environment-variable-only configuration for production, or at minimum documenting this limitation prominently. 4. Speculative model IDs in catalog ( The catalog includes model IDs that don't exist at the time of this PR (as of early 2026):
Including non-existent model IDs will fail silently at configuration time (they pass 5. return createProviderRegistry(providers as never);This completely disables TypeScript's type checking for the registry creation and hides any type incompatibilities between the provider objects and what P3 — Minor / UX Issues6. All 14 providers instantiated per extraction (
7. Silent candidate truncation ( When more than 220 candidates are found, the list is silently truncated (first 220 kept, rest dropped). There's a 8. admin: {
condition: (_, siblingData) => siblingData?.provider === "ollama",
...
},OpenAI-compatible providers and Azure OpenAI deployments commonly need custom base URLs. The 9. Ollama URL normalization is fragile ( const sanitized = baseURL.replace(/\/+$/, "");
return sanitized.endsWith("/v1") ? sanitized : `${sanitized}/v1`;This will incorrectly handle URLs like 10. If the model hits 40 steps without calling Nits
Summary: The architecture is solid and the backward compatibility path is well thought out. The two P1 issues (data loss during force re-extract and false-negative on all-chunk-failure) should be addressed before merge. The security concern around plaintext API key storage and the speculative model IDs in the catalog are also worth resolving or at least explicitly acknowledging. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4b1afc646b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…model enforcement - Add API key validation for AI providers with caching and timeout - Generate provider model IDs from installed AI SDK packages - Restrict extraction tasks to models with structured output + tool calling support - Reorganize Settings global into modular tab components (AI, Airtable, Meedan) - Add script to regenerate provider model IDs from type definitions - Improve provider credential configuration with real-time validation
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b7bf5b8cf3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ponent Move Settings.ts barrel export to direct import and create dedicated RowLabel component for AI provider rows. This improves component organization and enables custom row labeling in the admin UI. - Delete barrel export file to simplify imports - Add AIProviderRowLabel component for custom row display in array fields - Update admin import map to include new component - Configure AI credentials array to use custom RowLabel component - Refactor arrow function formatting for consistency
|
@claude Review |
|
Claude finished @kelvinkipruto's task in 2m 30s —— View job Code Review
This is a well-architected feature. The provider catalog/registry separation is clean, backward compatibility with legacy Google-only config is gracefully preserved, and the two-pass extraction with chunking is a meaningful improvement. Some issues from previous reviews are already addressed in the current code, which is great. Below are the remaining concerns. Status of Previously Flagged IssuesAlready addressed in the current code:
P1 — Data Loss1. Existing extractions deleted before new extraction completes ( On // Deletions happen here...
for (const extraction of existingExtractions) {
await payload.delete({ collection: "ai-extractions", id: extraction.id });
}
// But extraction can still fail below — no rollback possible
const extractedCandidates = await extractPromisesFromChunks({...});Fix: Move deletions to after 2. All-chunk failures silently become "No promises found" ( Per-chunk if (APICallError.isInstance(error) || NoObjectGeneratedError.isInstance(error)) {
continue; // ← all chunks could fail here with no signal
}Fix: After the chunk loop, if P2 — Security / UX3. API keys stored as plaintext ( The 4. Validation blocks saving when network is unavailable ( When if (status === "skipped") {
return `Could not verify API key for "${getProviderDisplayName(providerId)}". Check network/base URL and try again.`;
}Fix: Treat 5. Google API key exposed in URL query string ( const endpoint = `https://generativelanguage.googleapis.com/v1beta/models?key=${encodeURIComponent(apiKey)}`;Appending API keys as URL query parameters means they can appear in server access logs, browser history, and proxy logs. Prefer passing the key as a header (e.g. P3 — Minor / Design6. All 14 providers eagerly instantiated per extraction (
7. Identical implementations exist in 8. condition: (_, siblingData) => siblingData?.provider === "ollama",The 9. Fragile Ollama URL normalization ( const sanitized = baseURL.replace(/\/+$/, "");
return sanitized.endsWith("/v1") ? sanitized : `${sanitized}/v1`;
10. Silent candidate truncation not surfaced to users ( When >220 candidates are found, the list is silently truncated. The Nits
Summary: The two P1 issues (data loss during |
- Add reusable MaskedApiKeyField component for secure API key display in settings tabs - Extract trimToUndefined utility to shared module for consistent string handling - Improve Google API key validation to use header-based authentication - Enhance AI promise extraction to handle chunk failures and defer cleanup until successful re-extraction - Add warning about generated model IDs possibly not being deployed in provider accounts - Fix Ollama base URL normalization to properly handle path segments - Simplify provider registry by directly creating language models without intermediate registry
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b8790d23d7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@claude Review |
|
Claude finished @kelvinkipruto's task in 2m 24s —— View job Code Review
|
|
@claude Review |
|
Claude finished @kelvinkipruto's task in 2m 26s —— View job Code Review
This PR has significantly improved across review iterations. Most previously flagged P1 and P2 issues are now resolved. Below is the current status. Previously Flagged Issues — Now Resolved
Remaining IssuesP2 — API keys stored as
P2 — condition: (_, siblingData) => siblingData?.provider === "ollama",The Minor / NitsExample model ID in error message is a non-existent model ( return 'Use format "provider:model", for example "openai:gpt-5".';
Silent candidate truncation not stored in extraction record (
Dynamic model ID loading reads filesystem at module init (
No tests included — Summary: The architecture is solid and the codebase has improved substantially through the review cycle. The remaining items are a P2 security note about plaintext API key storage, a UI gap for non-Ollama |

Description
Fixes #538
Type of change
Screenshots
Checklist: