⚡ Bolt: Reduce worker thread head-of-line blocking#383
Conversation
Co-authored-by: AhmmedSamier <17784876+AhmmedSamier@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughReplaces fixed-size Promise.all chunking with a pLimit-based concurrency-limited task queue in the indexer worker and adds guidance recommending this pattern; also adds a standalone script that documents a suspected CI/build failure caused by an incorrect import path. ChangesIndexer worker refactor
Performance guidance
CI / Investigation script
Sequence Diagram(s)sequenceDiagram
actor Worker as Indexer Worker
participant Queue as pLimit Queue<br/>(BATCH_SIZE)
participant Parser as Parse Task
participant Accum as Result Accumulator<br/>(chunkItems)
participant Emitter as Result Emission
Worker->>Queue: enqueue filePath task
activate Queue
Queue->>Parser: run up to BATCH_SIZE parse tasks concurrently
activate Parser
Parser-->>Accum: append parsed SearchableItem(s)
deactivate Parser
alt processedCount reaches BATCH_SIZE
Accum->>Emitter: emit partial batch (count=BATCH_SIZE, isPartial=true)
end
alt All tasks complete
Accum->>Emitter: emit final batch (count=remainder, isPartial=false)
end
deactivate Queue
Estimated Code Review Effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Co-authored-by: AhmmedSamier <17784876+AhmmedSamier@users.noreply.github.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@language-server/src/core/indexer-worker.ts`:
- Around line 40-71: If filePaths is empty the Promise.all block never runs and
no terminal result is posted, leaking state.activeTasks; add an early check at
the top of the worker (in language-server/src/core/indexer-worker.ts) that if
filePaths.length === 0 immediately calls parentPort?.postMessage({ type:
'result', items: [], count: 0, isPartial: false }) and returns, so the worker
always emits a terminal result; this should be done before the Promise.all logic
that uses parser.parseFile, BATCH_SIZE, chunkItems and processedCount.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: a74c3287-48cc-4734-b27b-b8fdc27450a5
📒 Files selected for processing (2)
.jules/bolt.mdlanguage-server/src/core/indexer-worker.ts
Co-authored-by: AhmmedSamier <17784876+AhmmedSamier@users.noreply.github.com>
Co-authored-by: AhmmedSamier <17784876+AhmmedSamier@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
patch_last.js (1)
1-109: ⚡ Quick winDrop this one-off investigation script from the PR.
This file is an executable debugging transcript, not a test or a reusable build tool. Keeping it in the repo adds maintenance noise and stale context; if the narrative matters, it belongs in the PR description or an issue instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@patch_last.js` around lines 1 - 109, This file (patch_last.js) is a one-off debugging transcript and should be removed from the PR: delete patch_last.js from the repo/commit, ensure no other files import or reference it (search for "patch_last.js" and the top-level variables like content or the console.log("I see...") line), remove any executable bit or package.json script entries that might invoke it, and if the investigation needs to be preserved move the narrative into the PR description or an issue instead of keeping the script in source control.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@language-server/src/core/indexer-worker.ts`:
- Around line 30-36: The code uses message.chunkSize to set BATCH_SIZE and pass
into pLimit without validating it, which allows 0, negatives, NaN or non-integer
values to break scheduling and the `% BATCH_SIZE` flush/count math; add a guard
when computing BATCH_SIZE to coerce/validate message.chunkSize into a positive
integer (e.g., parse/Math.floor and fallback to a safe default like 25) and
ensure it is >= 1 before calling pLimit and using the modulo-based flush logic
so pLimit(BATCH_SIZE) and the count/% BATCH_SIZE calculations never receive
invalid values; update the assignment where BATCH_SIZE is defined and use that
validated constant in pLimit and the flush/count code paths.
---
Nitpick comments:
In `@patch_last.js`:
- Around line 1-109: This file (patch_last.js) is a one-off debugging transcript
and should be removed from the PR: delete patch_last.js from the repo/commit,
ensure no other files import or reference it (search for "patch_last.js" and the
top-level variables like content or the console.log("I see...") line), remove
any executable bit or package.json script entries that might invoke it, and if
the investigation needs to be preserved move the narrative into the PR
description or an issue instead of keeping the script in source control.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 8f92e501-7ed6-4063-a73e-e20faaa73bc7
📒 Files selected for processing (2)
language-server/src/core/indexer-worker.tspatch_last.js
…l resolution
When filePaths is empty (warmup path), the worker now silently returns
after initializing the parser instead of posting a result message.
The previous behavior posted { type: 'result', isPartial: false } for
empty batches. This message was queued before runFileIndexingPool
attached its listeners, but delivered during the actual indexing run.
The handler then decremented activeTasks prematurely, causing the pool
to resolve before any symbols were parsed — leaving the search index
with only file items and no symbols/types.
Fixes: 4 failing vscode integration tests (EVERYTHING, TYPES, SYMBOLS,
ENDPOINTS scopes all returning 0 results).
…https://github.com/AhmmedSamier/DeepLens into bolt/reduce-worker-hol-blocking-16728447728457543423
AhmmedSamier
deleted the
bolt/reduce-worker-hol-blocking-16728447728457543423
branch
1 participant
💡 What: Replaced a chunk-based
Promise.allwith a continuous concurrency pool (pLimit) inlanguage-server/src/core/indexer-worker.ts.🎯 Why: Previously, fast file parsing tasks had to wait for the slowest file in a fixed batch to complete before results were streamed back to the parent thread. This caused head-of-line blocking and unnecessary latency in the indexing pipeline.
📊 Impact: Improves parser throughput and provides smoother, faster streaming of code symbols back to the main thread by preventing fast tasks from stalling behind slow ones.
🔬 Measurement: Verify by executing the full test suite (
cd language-server && bun test) and observing the indexing speed when reloading the VS Code workspace. All 75 tests pass correctly.PR created automatically by Jules for task 16728447728457543423 started by @AhmmedSamier
Summary by CodeRabbit
Documentation
Improvements