Skip to content

optimizer: allow resume of short-completed runs #143

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ZhengyaoJiang wants to merge 1 commit into
base: dev
Choose a base branch
from
+22 −8
Open

optimizer: allow resume of short-completed runs #143

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 22 additions & 8 deletions weco/optimizer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,14 +437,6 @@ def resume_optimization(
console.print(f"[bold red]Error fetching run status: {e}[/]")
return False

run_status_val = status.get("status")
if run_status_val not in ("error", "terminated"):
console.print(
f"[yellow]Run {run_id} cannot be resumed (status: {run_status_val}). "
f"Only 'error' or 'terminated' runs can be resumed.[/]"
)
return False

objective = status.get("objective", {})
metric_name = objective.get("metric_name", "metric")
maximize = bool(objective.get("maximize", True))
Expand All @@ -455,6 +447,28 @@ def resume_optimization(
current_step = int(status.get("current_step", 0))
steps_remaining = int(total_steps) - current_step
Comment on lines 447 to 448
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot Apr 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard step parsing before checking resumable status

This change moved the resumable-status gate below current_step/total_steps parsing, so weco resume now attempts int(...) conversion even for runs that should be rejected immediately. If the API returns current_step or optimizer.steps as null (which can happen on older/incomplete run metadata), int(None) raises and the command crashes instead of printing the non-resumable status message; previously completed runs never hit this path because they were rejected first.

Useful? React with 👍 / 👎.


run_status_val = status.get("status")
# Allow resume for runs prematurely marked "completed" with current_step <
# total_steps. This happens when a transient `Failed to submit result` on
# the CLI side races with a successful backend ack — backend records the
# result and marks the run completed, but the CLI exits with submit_failed
# well short of the configured step budget. Treat these as resumable so
# users can pick up where they left off without losing their step history.
short_completed = run_status_val == "completed" and steps_remaining > 0
if run_status_val not in ("error", "terminated") and not short_completed:
console.print(
f"[yellow]Run {run_id} cannot be resumed (status: {run_status_val}, "
f"current_step={current_step}/{total_steps}). "
f"Resumable when status in (error, terminated) or "
f"completed with current_step < total_steps.[/]"
)
return False
if short_completed:
console.print(
f"[cyan]Run is marked completed at step {current_step}/{total_steps}. "
f"Resuming the remaining {steps_remaining} step(s).[/]"
)

model_name = (
(optimizer.get("code_generator") or {}).get("model") or (optimizer.get("evaluator") or {}).get("model") or "unknown"
)
Expand Down
Loading