Proxy external mode hardcodes native FC — no prompt-injection fallback
Problem
ProxyServer in external mode creates a LlamafileClient with mode="native" (proxy.py:169). There's no way to configure prompt-injected FC mode when connecting to an external backend.
This means the proxy only works with backends/models that support native tool calling via OpenAI format. Models without native FC support (or backends like vLLM serving models with incomplete tool-calling templates) can't use forge's prompt-injection fallback through the proxy.
In managed mode this isn't an issue — the eval runner and setup_backend() both accept mode and pass it through.
Proposed fix
Add --mode flag to forge-proxy CLI and mode parameter to ProxyServer.__init__(). Pass through to LlamafileClient(mode=...) in external mode.
Context
Came up from a user asking about vLLM + forge. vLLM serves OpenAI-compatible endpoints but not all models have native FC templates. Prompt-injection mode would make the proxy work with any OpenAI-compatible backend regardless of native FC support.
Proxy external mode hardcodes native FC — no prompt-injection fallback
Problem
ProxyServerin external mode creates aLlamafileClientwithmode="native"(proxy.py:169). There's no way to configure prompt-injected FC mode when connecting to an external backend.This means the proxy only works with backends/models that support native tool calling via OpenAI format. Models without native FC support (or backends like vLLM serving models with incomplete tool-calling templates) can't use forge's prompt-injection fallback through the proxy.
In managed mode this isn't an issue — the eval runner and
setup_backend()both acceptmodeand pass it through.Proposed fix
Add
--modeflag toforge-proxyCLI andmodeparameter toProxyServer.__init__(). Pass through toLlamafileClient(mode=...)in external mode.Context
Came up from a user asking about vLLM + forge. vLLM serves OpenAI-compatible endpoints but not all models have native FC templates. Prompt-injection mode would make the proxy work with any OpenAI-compatible backend regardless of native FC support.