[Bug]: EmbeddingStrategy mixes generator & embedding configs + leftover mock causes “fried rice” variations and 403s

[NickKotte]

crawl4ai version

0.7.6

Expected Behavior

Adaptive crawler. strategy="embedding",
embedding_model="openai/text-embedding-3-small", embedding_llm_config=LLMConfig( provider="openai/gpt-4o-mini", // openai/text-embedding-3-small fails here too api_token=OPENAI_API_KEY, ),
Query:
'capabilities, services,
certifications, description of work,
products'

Query Expansion:
Original query expanded to 12 variations

1. Where can I compare prices for various products?
2. What new products have been launched this year?
3. What products are recommended for pet owners?
4. What are the must-have products for outdoor activities?

Current Behavior

•	Query variations are replaced by a hard-coded “fried rice” list.
•	embedding_llm_config is reused for both generation and embeddings, so the wrong provider/model can hit the wrong API:
•	Chat model sent to embeddings endpoint → 403.
•	Embedding model used as a “provider” for text generation → failures or zero variations.
•	Embedding dimension sometimes mismatches the configured embedding_model.

Is this reproducible?

Yes

Inputs Causing the Bug

line 700
map_query_semantic_space function uses left over mock data
doesn't use the correct model for expansion or embedding

Steps to Reproduce

A) hard coded query variations:
1.	Use strategy="embedding" and call AdaptiveCrawler.digest(...).
2.	Observe variations list: always food-related (“fried rice…”) regardless of query.
B) 403 when embeddings are requested
`AdaptiveConfig(
  strategy="embedding",
  embedding_model="openai/text-embedding-3-small",
  embedding_llm_config=LLMConfig(
    provider="openai/gpt-4o-mini",
    api_token=OPENAI_API_KEY,
  ),
  n_query_variations=12,
)`
2.	Run digest(...).
3.	Intermittently see: `403 - You are not allowed to generate embeddings from this model` or end up with variations: 0, and embedding dims/behavior inconsistent with the configured model.

Code snippets

adaptive = AdaptiveCrawler(crawler, adaptive_cfg)
 result = await adaptive.digest(start_url=start_url, query=query)

Or literally just run the adaptive crawler example available in the Craw4ai repository.

OS

macOS

Python version

3.13.5

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

litellm.exceptions.BadRequestError: litellm.BadRequestError: OpenAIException - Error code: 403 - {'error': {'message': 'You are not allowed to generate embeddings from this model', 'type': 'invalid_request_error', 'param': None, 'code': None}}
and
Adaptive Crawl Stats - Query:
'capabilities, services,
certifications, description of work,
products'
Query Expansion:
Original query expanded to 4 variations

1. how to add flavor to vegetable fried rice?
2. what are the best vegetables to use in fried rice?
3. are there any tips for making healthy fried rice with vegetables?
4. how do I make vegetable fried rice from scratch?
   ...
   `

[matejcermak]

bump

[Ahmed-Tawfik94]

Observed Behavior

• Error output/stack trace (trimmed):

litellm.exceptions.BadRequestError: OpenAIException - Error code: 403 - {
  'error': {'message': 'You are not allowed to generate embeddings from this model', ...}
}

• Notes:
  • Query expansion sometimes returns unrelated “fried rice” variations in v0.7.6, regardless of user query.
  • Embeddings sometimes use the chat model name as the embedding model, causing 403s or wrong dimensions.

Expected Behavior

• Query expansion should use a chat/completions-capable model configured for generation.
• Embeddings should use the configured embedding model and not mix with the generation config.

Root Cause Analysis

• Primary cause:
  • EmbeddingStrategy is constructed with llm_config=self.config.embedding_llm_config, then uses that config for query expansion. This mixes embedding-only configuration with text generation. Evidence: crawl4ai/adaptive_crawler.py:1296-1299 and crawl4ai/adaptive_crawler.py:712-731.
• Contributing factors:
  • get_text_embeddings ignores the provided model_name when llm_config is supplied, using llm_config['provider'] as the embedding model instead. This sends chat models to the embedding endpoint and ignores embedding_model. Evidence: crawl4ai/utils.py:3573-3581.
  • In v0.7.6, map_query_semantic_space used leftover mock “fried rice” data, causing unrelated variations. The current main has these lines commented out, so this specific symptom appears resolved. Evidence: crawl4ai/adaptive_crawler.py:738-755 (mock data commented).
• Evidence:
  • crawl4ai/adaptive_crawler.py:1296-1299
  • crawl4ai/adaptive_crawler.py:712-731
  • crawl4ai/utils.py:3573-3581
  • crawl4ai/adaptive_crawler.py:738-755

[hafezparast]

Hi @NickKotte — thanks for reporting this! This has been fixed on the develop branch.

What was wrong: Three separate problems:

1. Hard-coded "fried rice" mock query variations were left in map_query_semantic_space (now commented out, real LLM generation is used)
2. embedding_llm_config was reused for both query expansion and embeddings, causing 403s when a chat model hit the embeddings endpoint
3. Embedding dimensions could mismatch the configured model

How it was fixed: Commit 0a45c10 added a separate query_llm_config field to AdaptiveConfig and EmbeddingStrategy, so query expansion uses a chat model and embeddings use the embedding model independently. A proper fallback chain ensures backward compatibility.

How to test

pip install git+https://github.com/unclecode/crawl4ai.git@develop

Then use the separated configs:

AdaptiveConfig(
    strategy="embedding",
    embedding_model="openai/text-embedding-3-small",
    embedding_llm_config=LLMConfig(
        provider="openai/text-embedding-3-small",
        api_token=OPENAI_API_KEY,
    ),
    query_llm_config=LLMConfig(
        provider="openai/gpt-4o-mini",
        api_token=OPENAI_API_KEY,
    ),
    n_query_variations=12,
)

Let us know if it resolves your issue!

@unclecode @ntohidi this can be closed.

[hafezparast]

Fix submitted in #1831 — uses the proper query_llm_config fallback chain in map_query_semantic_space, passes base_url/backoff params, and removes leftover mock data.

[hafezparast]

Hi @NickKotte — thanks for reporting this! A fix has been submitted in PR #1832.

What was wrong: Leftover commented-out mock data ("fried rice" variations) was still present in map_query_semantic_space. The config separation between query expansion and embeddings was already fixed on develop, but the mock data remained.
How it was fixed: Removed all commented-out mock data blocks. The function now cleanly uses _get_query_llm_config_dict() with proper fallback chain and passes base_url/backoff params.

How to test before it's merged

pip install git+https://github.com/hafezparast/crawl4ai.git@fix/maysam-mcp-ws-embedding-llm-1678-1574-1516

Then try your original reproduction case and let us know if it resolves your issue. Your feedback helps us merge with confidence!

[ntohidi]

This has been fixed and is now on the develop branch. It will be in the release and main branch soon.
I will close the issue, but feel free to continue the conversation or reopen it if the issue remains.