Skip to content

perf: increase default benchmark sizes for retrieval_core and graph_analytics #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Aaryan-Dadu merged 2 commits into from
Apr 12, 2026
Merged

perf: increase default benchmark sizes for retrieval_core and graph_analytics #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
81db385
fix : changed data size in retrieval_core and graph_analytics (fixes #2)
notdarking Apr 11, 2026
403c704
feat: make benchmark size configurable
notdarking Apr 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion chuck/__main__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,12 @@ def main() -> int:
],
help="Run a single capability benchmark",
)
bench_parser.add_argument(
"--size",
type=int,
default=None,
help="Override the default benchmark size"
)
Comment on lines +40 to +45
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Validate --size as a positive integer.

Line 42 currently accepts 0 and negative values, which can produce invalid benchmark payload sizes.

Proposed fix 
+def _positive_int(value: str) -> int:
+    parsed = int(value)
+    if parsed <= 0:
+        raise argparse.ArgumentTypeError("--size must be a positive integer")
+    return parsed
+
 ...
     bench_parser.add_argument(
-    "--size",
-    type=int,
-    default=None,
-    help="Override the default benchmark size"
+        "--size",
+        type=_positive_int,
+        default=None,
+        help="Override the default benchmark size",
     )
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@chuck/__main__.py` around lines 40 - 45, The --size argument currently allows
zero and negative values; change validation so only positive integers are
accepted by providing a custom argparse type (e.g., a function named
positive_int that parses int(x) and raises argparse.ArgumentTypeError if value
<= 0) and use it in bench_parser.add_argument("--size", type=positive_int, ...);
alternatively perform the same check after parsing and raise a clear error/exit
if args.size is not None and args.size <= 0. Ensure the unique symbol
bench_parser.add_argument("--size", ...) is updated to reference the validator
(or the post-parse check validates args.size).

Copy link
Copy Markdown
Member

@Aaryan-Dadu Aaryan-Dadu Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please show the demo for this

Copy link
Copy Markdown
Author

@notdarking notdarking Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot added in PR

subparsers.add_parser("regress", help="Run regression checks")
subparsers.add_parser("generate-baselines", help="Generate regression baselines")
snapshot_parser = subparsers.add_parser("snapshot", help="Create a performance/reliability snapshot")
Expand Down Expand Up @@ -81,7 +87,7 @@ def main() -> int:
command = args.command or "bench"

if command == "bench":
print(format_benchmarks(run_benchmarks(task=args.task)))
print(format_benchmarks(run_benchmarks(task=args.task,size=args.size)))
return 0
if command == "regress":
print(format_regression(run_regression()))
Expand Down
8 changes: 4 additions & 4 deletions chuck/benchmark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,12 @@
}


def run_benchmarks(task: str | None = None) -> list[dict[str, Any]]:
def run_benchmarks(task: str | None = None, size: int | None = None) -> list[dict[str, Any]]:
if task is None:
return [runner() for runner in RUNNERS]
runner = RUNNER_BY_NAME[task]
return [runner()]
return [runner(size=size) for runner in RUNNERS]

runner = RUNNER_BY_NAME[task]
return [runner(size=size)]

def _compact_output(output: Any) -> str:
if not isinstance(output, dict):
Expand Down
4 changes: 2 additions & 2 deletions chuck/benchmarks/compute_core/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
from ...tasks.compute_core import TASK_SPEC


def run() -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_009)
def run(size: int | None = None) -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_009,size=size)
Copy link
Copy Markdown
Member

@Aaryan-Dadu Aaryan-Dadu Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

returning size=None? will that work?

Copy link
Copy Markdown
Author

@notdarking notdarking Apr 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, In the benchmark_task function I have added a conditional statement if size is none it defaults to default benchmark_size

4 changes: 2 additions & 2 deletions chuck/benchmarks/data_encoding/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
from ...tasks.data_encoding import TASK_SPEC


def run() -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_004)
def run(size: int | None = None) -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_004,size=size)
4 changes: 2 additions & 2 deletions chuck/benchmarks/graph_analytics/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
from ...tasks.graph_analytics import TASK_SPEC


def run() -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_005)
def run(size: int | None = None) -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_005,size=size)
4 changes: 2 additions & 2 deletions chuck/benchmarks/io_pipeline/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
from ...tasks.io_pipeline import TASK_SPEC


def run() -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_001)
def run(size: int | None = None) -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_001, size=size)
Comment thread
notdarking marked this conversation as resolved.
4 changes: 2 additions & 2 deletions chuck/benchmarks/memory_index/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
from ...tasks.memory_index import TASK_SPEC


def run() -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_008)
def run(size: int | None = None) -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_008,size=size)
4 changes: 2 additions & 2 deletions chuck/benchmarks/memory_tier/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
from ...tasks.memory_tier import TASK_SPEC


def run() -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_007)
def run(size: int | None = None) -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_007,size=size)
4 changes: 2 additions & 2 deletions chuck/benchmarks/ordering_core/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
from ...tasks.ordering_core import TASK_SPEC


def run() -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_002)
def run(size: int | None = None) -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_002,size=size)
4 changes: 2 additions & 2 deletions chuck/benchmarks/prime_analytics/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
from ...tasks.prime_analytics import TASK_SPEC


def run() -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_006)
def run(size: int | None = None) -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_006,size=size)
4 changes: 2 additions & 2 deletions chuck/benchmarks/relational_fusion/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
from ...tasks.relational_fusion import TASK_SPEC


def run() -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_010)
def run(size: int | None = None) -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_010,size=size)
4 changes: 2 additions & 2 deletions chuck/benchmarks/retrieval_core/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
from ...tasks.retrieval_core import TASK_SPEC


def run() -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_003)
def run(size: int | None = None) -> dict[str, Any]:
return benchmark_task(TASK_SPEC, seed=1_003,size=size)
2 changes: 1 addition & 1 deletion chuck/tasks/graph_analytics/task.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,4 @@ def solve(graph: dict[str, list[str]], iterations: int = 16, damping: float = 0.
}


TASK_SPEC = TaskSpec("graph_analytics", generate, solve, 48, 1_000)
TASK_SPEC = TaskSpec("graph_analytics", generate, solve, 48, 200_000)
Comment thread
Aaryan-Dadu marked this conversation as resolved.
2 changes: 1 addition & 1 deletion chuck/tasks/retrieval_core/task.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ def solve(payload: dict[str, Any]) -> dict[str, Any]:
generate,
solve,
48,
2_000,
2000_000,
Comment thread
Aaryan-Dadu marked this conversation as resolved.
algorithm_style="probabilistic",
reliability_floor=0.88,
)
Loading