Connection to Ollama not accepting models

[Ossenbrugge]

Hi,

This is a fantastic tool. I'm having issues getting Ollama to work. When I select it as a provider, the models that are pulled in Ollama are showing, in Director's Console. However, when I select one it isn't saved. Additionally, trying to use the LLM to enhance a prompt does not work. I get this error: Network error: 405, message='Attempt to decode JSON with unexpected mimetype: text/plain', url='http://192.168.8.122:11434'

Any help (I understand this may be PEBKAC) is appreciated. Thanks!

[Quipjoy]

Hi! Just wanted to say thanks for this project — Awesome work, and thanks for sharing it!
same issue for me here, i have temporary fix it in this way, hope help you to fix in official rel.

Bug Fix Report for Ollama Integration

This report details fixes for issues encountered when using the Ollama provider in the Directors Console.

Issues Addressed

1. Ollama Connection Error (405 Method Not Allowed): The application was sending requests to http://localhost:11434 instead of the chat endpoint.
2. Model Name Mismatch: The application was sending model names like ollama:llama3 to Ollama, but Ollama expects llama3.
3. Invalid Model Selection: Embedding models (e.g., nomic-embed-text) were appearing in the chat model selection list, leading to errors.
4. Settings Persistence Failure: Selecting models with colons in their ID (like llama3:latest) caused the selection to be lost upon page reload due to incorrect parsing in the frontend.

Changes

Backend: api/providers/llm_service.py

Fix _call_local method

• Endpoint Correction: Explicitly append /api/chat to the Ollama endpoint.
• Model Name Sanitization: Strip the ollama: prefix from the model name before sending the request.

if provider == "ollama":
    # Append /api/chat if not present
    if not endpoint.endswith("/api/chat"):
        endpoint = f"{endpoint}/api/chat"
    
    # Strip provider prefix if present
    if model.startswith("ollama:"):
        model = model.split(":", 1)[1]

Fix _fetch_local_models method

• Filter Embedding Models: Add logic to exclude models with "embed" or "embedding" in their name from the list.

if provider == "ollama":
    for model in data.get("models", []):
        name = model.get("name", "")
        # Filter out embedding models
        if "embed" in name.lower() or "embedding" in name.lower():
            continue
        # ... append to models list

Frontend: frontend/src/components/Settings.tsx

Fix loadLlmSettingsFromServer method

• Correct Model ID Parsing: Update the parsing logic to correctly handle model IDs that contain colons. Previously, splitting by : would break IDs like llama3:latest.

// Old erroneous code:
// const [provider, model] = settings.selected_model.includes(':') 
//   ? settings.selected_model.split(':')
//   : ['', settings.selected_model];

// New correct implementation:
const firstColonIndex = settings.selected_model.indexOf(':');
if (firstColonIndex !== -1) {
  return {
    provider: settings.selected_model.substring(0, firstColonIndex),
    model: settings.selected_model.substring(firstColonIndex + 1)
  };
}

Note on Timeout

The default timeout for LLM requests is set to 60 seconds. For local LLMs on slower hardware, this might need to be increased if "Request timed out" errors persist.