1.Knowledge Graph Building Pipeline

[santo0]

Knowledge Graph Pipeline for Query Difficulty Estimation

Introduces a fully modular knowledge graph (KG) construction pipeline under src/knowledge_graph/. The KG is intended to support query difficulty estimation by representing corpus concepts and their relationships as a graph and for graph-based retrieval. See src/knowledge_graph/README.md for detailed information about this module.

PRs structure

The PRs depend on the previous ones.

1. Current, start here
2. 2.Canonicalization, Section Tree and KG Retriever #98
3. 3.Summary Tree & Retriever Integration #105

PR Review — Knowledge Graph: Extraction Cache & Extractor Selection

Key Changes

build.py

• Constants and paths for graph building support
• load_chunks method for loading chunks and meta pkl generated by index builder.

llm_extract_keywords.py

• Run OpenRouterExtractor, for extracting the keywords for all chunks and store the result in a json, so later we can skip the Extractor step when generating the graph.

run_kg_pipeline.py

• Run keyword graph building process.

pipeline.py

• Calls the components for generating and persisting the graph.

extractors

• Folder with the extractors classes
  1. BaseExtractor: Interface for all extractors
  2. JsonExtractor: Returns given json extractor results (generated in llm_extract_keywords.py, for example)
  3. KeyBERTExtractor: Runs KeyBERT for extracting KW
  4. OpenRouterExtractor: Runs OpenRouter LLM (you can select which model to use) with given API key.
  5. SLMExtractor: Runs local Small Language Model (given .gguf file).

linkers

• Only one linker (ATM), Co-occurrence linker, which links keywords if two keywords appear in the same chunk. Each link has a weight corresponding on how many chunks both keywords appear at the same time.

pipeline.py

• Calls the components for generating and persisting the graph.

Other files

• prompts.py: The prompts used in the LLMs calls.
• openrouter_client.py: Client used for interacting with OpenRouter, required API key.
• models.py: Stores the models used in the KG code.

---

Notable Design Decisions

• LLM cache is append-only: old extraction files are never deleted; last write wins via symlink.
  No merging of partial runs.
• Live extractors don't write to cache: running --extractor openrouter via the pipeline
  extracts inline without updating extractions/latest.json. Use llm_extract_keywords.py
  directly if you want to cache the results first.

[santo0]

I'm trying to simplify this PR, I will notify when I'm done.