Skip to content

[NEW_PATTERN]_{20}_{SAME_PATCH_DIFFERENT_TOOL}. #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
buabeng2 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[NEW_PATTERN]_{20}_{SAME_PATCH_DIFFERENT_TOOL}. #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions graph_analysis/analyzer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -510,6 +510,7 @@ def get_patch_summary(self):
- fail_types: breakdown of all failure types encountered.
- fail_streaks: dict with max, average, and count of consecutive failed patch attempts.
- flip_flop: True if an edit is undone by a reverse change.
- same_patch_different_tool: Ocurrence of the same edit being attempted with different tools (e.g. str_replace → manual patch).
- repeat_failed_edit: True if a previously failed patch is attempted and failed again.
- abandonment: True if there exists a file with ≥1 attempts and no success on that file).
- fail_to_success_patterns: common reasoning phase transitions from a failed to successful patch.
Expand Down Expand Up @@ -540,6 +541,7 @@ def get_patch_summary(self):
seen_edits = set()
flip_flop = False
repeat_failed_edit = False
same_patch_different_tool = False
abandonment = False
edit_history = []
current_streak = 0
Expand All @@ -551,6 +553,7 @@ def get_patch_summary(self):
previous_status = None
previous_edit = None
previous_step = None
prev_label = None

# --- track success/failure per file path to detect "abandonment on some path" ---
file_attempts = Counter() # path -> #attempts
Expand All @@ -576,6 +579,7 @@ def get_patch_summary(self):
new_str = args.get("new_str", "") if isinstance(args, dict) else ""
status_raw = args.get("edit_status", "") if isinstance(args, dict) else ""
edit_key = (path, old_str, new_str)
label = data.get("label", "")

# Count per-file attempts
if path:
Expand Down Expand Up @@ -603,6 +607,10 @@ def get_patch_summary(self):
# undo_edit
if data.get("label", "") == "str_replace_editor: undo_edit":
flip_flop = True
# same edit different tool check
if edit_history and old_str == edit_history[-1][1] and new_str == edit_history[-1][2] and path == edit_history[-1][0] and label != prev_label:
same_patch_different_tool = True


# Reasoning transitions from fail -> success
if previous_status != "success" and status == "success":
Expand All @@ -623,6 +631,7 @@ def get_patch_summary(self):
previous_edit = edit_key
previous_status = status
previous_step = step
prev_label = label

if current_streak > 0:
fail_streaks.append(current_streak)
Expand Down