Usage example of SequenceR on any other benchmark

[jose]

Hi @ASSERT-KTH,

Any chance you could provide a quick step-by-step on how to run SequenceR on any other benchmark than Defects4J? For instance, Bears or Bugs.jar.

Thanks in advance.

[monperrus]

Hi @jose

Thanks for reaching out.

The doc says

$ ./sequencer-predict.sh --model=[model path] --buggy_file=[abs path] --buggy_line=[int] --beam_size=[int] --output=[abs path]

Where the buggy file can be any arbitrary Java file.

cc/ @chenzimin

[jose]

Thanks @monperrus, somehow I missed that.

(Note for future self, --beam_size has been initialised with the value 50 for Defects4J).

Two questions:

1. How would one initialise the --buggy_line parameter on faults of omission, i.e., the fix did not modified any existing code, it added new code instead to fix the buggy behaviour, for example, Chart-14?
2. Do you, by any chance, know where I can find the set buggy lines of each defect on Bears, Bugs.jar, IntroClassJava, and QuixBugs?

[monperrus]

How would one initialise the --buggy_line parameter on faults of omission

• In the Sequencer paper, we did not consider those, we only considered replacement bugs.
• In general, fault localization of omission bugs is rather under-researched.

Do you, by any chance, know where I can find the set buggy lines of each defect on Bears, Bugs.jar, IntroClassJava, and QuixBugs?

Do you mean the ground truth line of the ground truth developer patch?

[jose]

In the Sequencer paper, we did not consider those, we only considered replacement bugs.

Got it.

In general, fault localization of omission bugs is rather under-researched.

It's indeed. FYI, @speezepearson, @rjust, @mernst, @ruimaranhao, @gofraser and I have identified for every fault of omission in D4J (v1.4.0?), the set of lines present in the buggy version that could perhaps be blamed of the buggy behaviour. You may find them in here. We used that data in the paper Evaluating and improving fault localization.

Do you mean the ground truth line of the ground truth developer patch?

Yes.

[andre15silva]

Hi @jose!

I usually load these from the ground truth patch.
The patch contains all the information required (file path, added/removed lines).

Some benchmarks (e.g. Defects4J) provide the patch file (even though inverted), but for others you have to compute it yourself (e.g. Bears, QuixBugs).
I have the code to do this:

• Bears (uses my fork: https://github.com/andre15silva/bears-benchmark): https://github.com/andre15silva/mufin/blob/01ac2ff7c63a763802b4e0c0a9c0acad02e1d523/src/models/bears/bears.py#L133-L181
• QuixBugs (uses my fork: https://github.com/andre15silva/QuixBugs): https://github.com/andre15silva/mufin/blob/01ac2ff7c63a763802b4e0c0a9c0acad02e1d523/src/models/quixbugs/quixbugs.py#L27-L55
• Bugs.Jar: https://github.com/andre15silva/mufin/blob/01ac2ff7c63a763802b4e0c0a9c0acad02e1d523/src/models/bugsdotjar/bugsdotjar.py#L28-L62

Hopefully this helps :)

[jose]

Thanks @andre15silva.

I failed to see, from your pointers, how could I get the set of buggy lines of a given bug in either Bears, Bugs.jar, IntroClassJava, and QuixBugs. As far I understood, that code only checkouts all bugs of each benchmark, right?

I do have a script to collect the set of buggy lines and you can find it here. It is tied to Defects4J but it could be easily adapted for any other benchmark. My original question was whether the set buggy lines of each defect in Bears, Bugs.jar, IntroClassJava, and QuixBugs was available somewhere so that I (or anyone else) don't have to compute that.

@monperrus, back to the command to execute SequenceR

$ ./sequencer-predict.sh --model=[model path] \
  --buggy_file=[abs path] \
  --buggy_line=[int] \
  --beam_size=[int] \
  --output=[abs path]

How would I run the script on a defect with multiple faulty lines? Run it as many times as the number of faulty lines?

[andre15silva]

Sorry @jose , I wasn't explicit enough.

I don't have the set buggy lines you want.
What I meant is that I usually get these at runtime from the developer's patch, since the patch includes all the required information (file paths, added/removed lines) to obtain them.

This is straight-forward in Defects4J because the diff files are stored in the repo, but other benchmarks don't have this.
That is what my code does, compute the diff: 1) checkout the buggy and fixed versions; 2) do some processing if needed; 3) call the utils.get_diff() which just computes a diff between both versions.

Since different benchmarks have different ways of storing bugs, and to get a clean diff you sometimes need to make some subtle changes (like changing the package names in QuixBugs), I thought that code could be useful for you.

[jose]

This is straight-forward in Defects4J because the diff files are stored in the repo

Kinda. :-)
Defects4J provides minimal bug-introducing patches, not fixing patches.

but other benchmarks don't have this.

I found that Bugs.jar also provides, out-of-the-box, the developer patch, e.g., in here. Additionally,

• For IntroClassJava there are no fixed versions, right?
• Bears and QuixBugs do not provide that metadata but one could compute that, as you do.

call the utils.get_diff() which just computes a diff between both versions.

I completely missed that, sorry. (Note to future self, the command to compute the diff is in here.)

Thanks @andre15silva, I really appreciate your help.

[andre15silva]

Kinda. :-) Defects4J provides minimal bug-introducing patches, not fixing patches.

Yes, the reverse diff!

I found that Bugs.jar also provides, out-of-the-box, the developer patch, e.g., in here.

Oh, didn't see that before.

* For [IntroClassJava](https://github.com/Spirals-Team/IntroClassJava) there are no fixed versions, right?

Not AFAIR. I think I never used it because of that actually.

I completely missed that, sorry. (Note to future self, the command to compute the diff is in here.)

Thanks @andre15silva, I really appreciate your help.

No worries! Be aware that the command to compute the diff is tailored to that project (e.g. context window), you might want to tweak it yourself.

[monperrus]

IntroClassJava provides the reference implementation. So you can diff the buggy version and the reference implementation, it gives you a patch, but it's an "artificial" one. Might be useful for some analyses though.