tabular data ingestion#1720

Merged

jperez999 merged 119 commits intoNVIDIA:mainfrom

ftatiana-nv:staging

Apr 7, 2026

Commits on Mar 25, 2026

add ingest_structured entry point
tomer-levin-nv
committed
structured methods placeholders for inprocess ingestor
tomer-levin-nv
committed
run ingest structured as future task
tomer-levin-nv
committed
neo4j

liavnave
authored and
tomer-levin-nv
committed
DuckDB and spider

liavnave
authored and
tomer-levin-nv
committed
fix

liavnave
authored and
tomer-levin-nv
committed
DuckDB fixes

liavnave
authored and
tomer-levin-nv
committed
use neo_con from the store
tomer-levin-nv
committed
fix

DinaLaptii
authored and
tomer-levin-nv
committed
fix file

DinaLaptii
authored and
tomer-levin-nv
committed
Revert "fix"

DinaLaptii
authored and
tomer-levin-nv
committed
adding db population files

polazilber
authored and
tomer-levin-nv
committed
Add description_suggestion module (description_dal, functions, system_prompts)

ftatiana-nv
authored and
tomer-levin-nv
committed
update the uses of neo4j and duckdb

polazilber
authored and
tomer-levin-nv
committed
remove bi

ftatiana-nv
authored and
tomer-levin-nv
committed
add neo4j txt retrieval for embedding

ftatiana-nv
authored and
tomer-levin-nv
committed
trying to run

ftatiana-nv
authored and
tomer-levin-nv
committed
fixes to db retrieval funcs

ftatiana-nv
authored and
tomer-levin-nv
committed
fix imports

ftatiana-nv
authored and
tomer-levin-nv
committed
Merge remote-tracking branch 'origin/add-population-files' into fix/fix-duckdb_engine
tomer-levin-nv
committed
fix duckdb
tomer-levin-nv
committed
ingest_structured pipeline
tomer-levin-nv
committed
clean population

ftatiana-nv
authored and
tomer-levin-nv
committed
extract data and docker neo4j compose
tomer-levin-nv
committed
prepare for embeddings
tomer-levin-nv
committed
remove coalesce

ftatiana-nv
authored and
tomer-levin-nv
committed
init embed and vdb params
tomer-levin-nv
committed
remove unused functions from prepare_embedding_text.py

polazilber
authored and
tomer-levin-nv
committed
rename node labels to PascalCase and :schema relationship to :CONTAINS

polazilber
authored and
tomer-levin-nv
committed
add lance db table for structured

ftatiana-nv
authored and
tomer-levin-nv
committed
Add SQL tool with DuckDB connection and Deep Agent integration

polazilber
authored and
tomer-levin-nv
committed
Add langchain-nvidia-ai-endpoints, langchain-community, and deepagents dependencies

polazilber
authored and
tomer-levin-nv
committed
Add generate_sql step wired through Retriever
tomer-levin-nv
committed
Add top-k retrieval-augmented prompt for SQL agent

polazilber
authored and
tomer-levin-nv
committed
Add top-k retrieval-augmented prompt for SQL agent

polazilber
authored and
tomer-levin-nv
committed
do not end empty description to txt

ftatiana-nv
authored and
tomer-levin-nv
committed
add load dotenv

ftatiana-nv
authored and
tomer-levin-nv
committed
add json parser

ftatiana-nv
authored and
tomer-levin-nv
committed
add missing deep agent skills

polazilber
authored and
tomer-levin-nv
committed
fix generate_sql pipeline

polazilber
authored and
tomer-levin-nv
committed
Retry loop removed in neo4j store

ftatiana-nv
authored and
tomer-levin-nv
committed
Unify with Neo4jConnectionManager with Neo4jConnection.

ftatiana-nv
authored and
tomer-levin-nv
committed
remove temp schema parsing

ftatiana-nv
authored and
tomer-levin-nv
committed
remove log related lists

ftatiana-nv
authored and
tomer-levin-nv
committed
store -> neo4jconnection

ftatiana-nv
authored and
tomer-levin-nv
committed
Refactor structured ingestion: split extraction, remove PII, sync interface
tomer-levin-nv
committed
refactor structured_data

polazilber
authored and
tomer-levin-nv
committed
run embedding remotely

ftatiana-nv
authored and
tomer-levin-nv
committed
move file

ftatiana-nv
authored and
tomer-levin-nv
committed
feat: add dedicated structured ingest pipeline
tomer-levin-nv
committed
get structured output from llm

ftatiana-nv
authored and
tomer-levin-nv
committed
removed not relevant info

ftatiana-nv
authored and
tomer-levin-nv
committed
remove unused

ftatiana-nv
authored and
tomer-levin-nv
committed
rename engine to connectors

ftatiana-nv
authored and
tomer-levin-nv
committed
feat: move structured ingest pipeline to batch mode
tomer-levin-nv
committed
gitignore

ftatiana-nv
authored and
tomer-levin-nv
committed
changes to debug file

ftatiana-nv
authored and
tomer-levin-nv
committed
add top k

ftatiana-nv
authored and
tomer-levin-nv
committed
All labels should be taken from labels enum.

ftatiana-nv
authored and
tomer-levin-nv
committed
changes from the branch

ftatiana-nv
authored and
tomer-levin-nv
committed
refactor

ftatiana-nv
authored and
tomer-levin-nv
committed
remove file

ftatiana-nv
authored and
tomer-levin-nv
committed
CR fixes

liavnave
authored and
tomer-levin-nv
committed
fix: unpack queries from create_dataframe and fix Cypher commas in schemas_dal
tomer-levin-nv
committed
fix: rename embedding_api_key to api_key, update EmbedParams validation and ExtractParams defaults
tomer-levin-nv
committed
refactor: use __name__ for logger names across relational_db graph module
tomer-levin-nv
committed
fix: remove duplicate structured pipeline methods and unused imports in ingestor.py
tomer-levin-nv
committed
style: fix pre-commit issues (end-of-file newlines, black formatting, comment spacing)
tomer-levin-nv
committed
refactor: rename structured → tabular across relational DB pipeline
tomer-levin-nv
committed
refactor: rename populate_structured_data → populate_tabular_data
tomer-levin-nv
committed
refactor: rename relational_db → tabular_data and reorganize into connectors/ingestion/retrieval/neo4j
tomer-levin-nv
committed
refactor: introduce Edges const class and use Labels/Edges in db_dal Cypher queries
tomer-levin-nv
committed
refactor: flatten ingestion/population/ into ingestion/ and move prepare_embedding_text sibling
tomer-levin-nv
committed
refactor(tabular_data): simplify ingestion DAL and graph model

liavnave
authored and
tomer-levin-nv
committed
refactor(tabular_data): move Spider2 dev tools out of src

liavnave
authored and
tomer-levin-nv
committed
fix(duckdb): make connection param required (non-optional)

liavnave
authored and
tomer-levin-nv
committed
refactor: merge embedding helpers into fetch_tabular_embedding_dataframe and remove debug file
tomer-levin-nv
committed
update neo4j

yuvalshkolar
authored and
tomer-levin-nv
committed
refactor(tabular_data): remove account_id from graph indexes and clean up schema ingestion
tomer-levin-nv
committed
refactor(schemas_dal): remove unused column fields from graph query
tomer-levin-nv
committed
refactor(schemas_dal): remove deleted-node filters from all schema queries

liavnave
authored and
tomer-levin-nv
committed
refactor(tabular_data): remove table_type, row_count, size, retention_time, last_altered, default, length, scale from table/column models
tomer-levin-nv
committed
fix connector

yuvalshkolar
authored and
tomer-levin-nv
committed
refactor(tabular_data): remove unused added_or_modified_tables tracking
tomer-levin-nv
committed
refactor(indexes): remove unused fulltext name_index
tomer-levin-nv
committed
format

yuvalshkolar
authored and
tomer-levin-nv
committed
refactor(tabular_data): rename load_* to normalize_* and extract_relational_db_data to extract_tabular_db_data
tomer-levin-nv
committed
refactor(tabular_data): rename populate_data.py to write_to_graph.py
tomer-levin-nv
committed
pyproject

yuvalshkolar
authored and
tomer-levin-nv
committed
Merge branch 'main' into staging
yuvalshkolar
authored

Commits on Mar 30, 2026

Commits on Mar 31, 2026

Commits on Apr 6, 2026