[Enhancement] Fulltext index BloomFilter pushdown pre-filter#23833
Merged
mergify[bot] merged 8 commits intomatrixorigin:mainfrom Mar 23, 2026
Merged
[Enhancement] Fulltext index BloomFilter pushdown pre-filter#23833mergify[bot] merged 8 commits intomatrixorigin:mainfrom
mergify[bot] merged 8 commits intomatrixorigin:mainfrom
Conversation
…rigin#23832) Implement BloomFilter pushdown for fulltext index tables, similar to the existing IVF index pushdown. When a fulltext search query has additional filter predicates (e.g. WHERE match(...) against(...) AND category = 'tech'), the filtered PKs are collected into a BloomFilter and pushed down to the fulltext index table reader to skip irrelevant rows early. Key changes: - plan: applyJoinFullTextIndices builds RuntimeFilterBuildList with BF - compile: scope.go/remoterun.go pass BF from context to FilterHint - reader: tryUpdateColumns uses doc_id column for BF filtering on fulltext tables (independent of pkPos since internal SQL doesn't include __mo_fake_pk_col) - pk_filter_mem: NewMemPKFilter sets BFSeqNum using doc_id column for fulltext tables (fallback when __mo_index_pri_col is absent) - build_ddl: fix fulltext index table relkind from SystemIndexRel to FullTextIndex_TblType so TableType check matches at runtime Approved by: @xxx Ref matrixorigin#23832
ck89119
requested review from
XuPeng-SH,
aunjgr,
fengttt,
heni02 and
ouyuanning
as code owners
cpegeric
approved these changes
Mar 13, 2026
heni02
approved these changes
Mar 13, 2026
aunjgr
approved these changes
Mar 17, 2026
…lkind 1. Apply BloomFilter probe to all fulltext table function nodes instead of only the first one when multiple fulltext_match expressions exist. 2. Add IsFullTextIndexTableType() to handle legacy fulltext index tables whose relkind is still 'i' (SystemIndexRel) instead of 'fulltext', so BF pushdown works for pre-upgrade tables as well.
XuPeng-SH
approved these changes
Mar 23, 2026
Contributor
Merge Queue Status
This pull request spent 56 minutes 3 seconds in the queue, including 55 minutes 51 seconds running CI. Required conditions to merge
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Which issue(s) this PR fixes:
issue #23832
What this PR does / why we need it:
Implement BloomFilter pushdown for fulltext index tables, similar to the existing IVF index pushdown.
When a fulltext search query has additional filter predicates (e.g.
WHERE match(...) against(...) AND category = 'tech'), the filtered PKs are collected into a BloomFilter and pushed down to the fulltext index table reader to skip irrelevant rows early.current
applyJoinFullTextIndices's query plan：When there is a filter, the doc_id returned by fulltext_index_scan can be massively selected for filters, resulting in wastes significant I/O.
optimized query plan(with filter)
Key changes:
plan (
apply_indices_fulltext.go):applyJoinFullTextIndicesbuildsRuntimeFilterBuildListwith BloomFilter for the inner join between fulltext results and the source table.compile (
scope.go,remoterun.go): Pass BloomFilter from context toFilterHintwhen the target table is a fulltext index table (TableType == "fulltext").reader (
reader.go):tryUpdateColumnsusesdoc_idcolumn for BF filtering on fulltext tables, independent ofpkPossince internal SQL does not include__mo_fake_pk_col.pk_filter_mem (
pk_filter_mem.go):NewMemPKFiltersetsBFSeqNumusingdoc_idcolumn for fulltext tables as fallback when__mo_index_pri_colis absent.build_ddl (
build_ddl.go): Fix fulltext index tablerelkindfromSystemIndexRel("i") toFullTextIndex_TblType("fulltext") so theTableTypecheck matches at runtime.Bug fixes included:
buildFullTextIndexTablewrote"i"tomo_tables.relkindinstead of"fulltext", causing BF context propagation to be skipped.if pkPos != -1block, but fulltext internal SQL never includes the PK column.NewMemPKFilteronly looked for__mo_index_pri_colto setBFSeqNum; fulltext tables usedoc_idinstead.Tests:
NewMemPKFilterwith fulltext table BFSeqNumtryUpdateColumnswith fulltext BF filteringtest/distributed/cases/fulltext/fulltext_pushdown.sqlwith 7 scenarios