Fully auto-generate rerun examples#908
Open
blooop wants to merge 2 commits into
Open
Conversation
Delete example_rerun.py (replaced by generated capture_window), example_rerun2.py (niche utility demo), and example_rerun_over_time.py (support module). Inline ControlSystemSweep via inspect.getsource() in the meta-generator so sweep and regression examples are self-contained.
Contributor
Reviewer's GuideAuto-generates rerun examples so they are self-contained, inlining the ControlSystemSweep class into generated files and updating tooling and references to rely on the generated examples instead of deleted hand-written ones. Class diagram for rerun meta generator and ControlSystemSweep inliningclassDiagram
class MetaGeneratorBase {
+output_dir str
+template_filename str
+generate_example_file(function_name, imports, body, class_code, run_kwargs)
}
class MetaRerun {
+benchmark()
+_generate_regression()
+_generate_sweep()
}
MetaGeneratorBase <|-- MetaRerun
class ControlSystemSweep {
+to_bench(run_cfg)
+benchmark(bench_cfg)
}
class GeneratedExampleRerunRegression {
+example_rerun_regression(run_cfg)
+ControlSystemSweep inlined
}
class GeneratedExampleRerunSweep {
+example_rerun_sweep(run_cfg)
+ControlSystemSweep inlined
}
MetaRerun ..> ControlSystemSweep : imports_source
MetaRerun ..> GeneratedExampleRerunRegression : generates
MetaRerun ..> GeneratedExampleRerunSweep : generates
Flow diagram for ControlSystemSweep source inlining into generated rerun examplesflowchart TD
A[Start rerun example generation] --> B[Call _get_control_system_class_code]
B --> C["Try inspect.getsource(ControlSystemSweep)"]
C -->|success| D[Return ControlSystemSweep source string]
C -->|OSError| E[Resolve example_rerun_sweep.py with importlib.resources.files]
E --> F[Read file text with UTF-8 encoding]
F --> G[Return ControlSystemSweep source string from file]
D --> H[Pass class_code to generate_example_file]
G --> H[Pass class_code to generate_example_file]
H --> I[Write self-contained generated rerun example file]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The generator now depends on
ControlSystemSweepfrom a generated module at import time; consider movingControlSystemSweepinto a non-generated, stable module to avoid bootstrapping or circular-dependency issues when regenerating examples or changing generated paths. - In
_get_control_system_class_code, the importlib fallback currently reads the entireexample_rerun_sweep.pyfile rather than just theControlSystemSweepdefinition; if the file grows or gains additional top-level code this may produce incorrect inlined output, so it may be safer to parse and extract only the class body.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The generator now depends on `ControlSystemSweep` from a generated module at import time; consider moving `ControlSystemSweep` into a non-generated, stable module to avoid bootstrapping or circular-dependency issues when regenerating examples or changing generated paths.
- In `_get_control_system_class_code`, the importlib fallback currently reads the entire `example_rerun_sweep.py` file rather than just the `ControlSystemSweep` definition; if the file grows or gains additional top-level code this may produce incorrect inlined output, so it may be safer to parse and extract only the class body.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Performance Report for
|
| Metric | Value |
|---|---|
| Total tests | 1254 |
| Total time | 109.90s |
| Mean | 0.0876s |
| Median | 0.0020s |
Top 10 slowest tests
| Test | Time (s) |
|---|---|
test.test_bench_examples.TestBenchExamples::test_example_meta |
21.595 |
test.test_over_time_save_perf::test_save_faster_without_aggregated_tab |
5.344 |
test.test_hash_persistent.TestCrossProcessDeterminism::test_hash_stable_across_two_processes[ResultBool] |
4.037 |
test.test_over_time_repeats.TestMaxSliderPoints::test_default_subsampling_caps_at_max |
3.168 |
test.test_generated_examples::test_generated_example[cartesian_animation/example_cartesian_animation.py] |
3.083 |
test.test_generated_examples::test_generated_example[result_types/result_image/example_result_image_to_video.py] |
2.861 |
test.test_bencher.TestBencher::test_combinations_over_time |
1.448 |
test.test_optuna_result.TestOptunaResult::test_collect_optuna_plots_with_repeats |
1.050 |
test.test_bench_runner.TestBenchRunner::test_bench_reuse_report_cleared |
1.017 |
test.test_over_time_repeats.TestMaxSliderPoints::test_no_subsampling_when_below_default_max |
0.938 |
Updated by Performance Tracking workflow
Performance Report for
|
| Metric | Value |
|---|---|
| Total tests | 1254 |
| Total time | 105.41s |
| Mean | 0.0841s |
| Median | 0.0020s |
Top 10 slowest tests
| Test | Time (s) |
|---|---|
test.test_bench_examples.TestBenchExamples::test_example_meta |
21.055 |
test.test_over_time_save_perf::test_save_faster_without_aggregated_tab |
5.152 |
test.test_hash_persistent.TestCrossProcessDeterminism::test_hash_stable_across_two_processes[ResultBool] |
3.780 |
test.test_generated_examples::test_generated_example[cartesian_animation/example_cartesian_animation.py] |
3.037 |
test.test_over_time_repeats.TestMaxSliderPoints::test_default_subsampling_caps_at_max |
2.931 |
test.test_generated_examples::test_generated_example[result_types/result_image/example_result_image_to_video.py] |
2.814 |
test.test_bencher.TestBencher::test_combinations_over_time |
1.417 |
test.test_optuna_result.TestOptunaResult::test_collect_optuna_plots_with_repeats |
1.024 |
test.test_bencher.TestBencher::test_combinations |
0.901 |
test.test_over_time_repeats.TestMaxSliderPoints::test_no_subsampling_when_below_default_max |
0.896 |
Updated by Performance Tracking workflow
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
example_rerun.py,example_rerun2.py,example_rerun_over_time.py)ControlSystemSweepclass into generated examples viainspect.getsource(), matching the pattern used by the performance generatorexample_rerun_sweep.pyandexample_rerun_regression.pyare now self-containeddemo_rerunpixi task to point at the generated capture_window exampleTest plan
pixi run python bencher/example/meta/generate_meta_rerun.pysucceeds🤖 Generated with Claude Code
Summary by Sourcery
Auto-generate self-contained rerun examples and remove legacy hand-written ones.
Enhancements:
Chores: