Bug fixing with LLMs - #900
Open
jodavies wants to merge 3 commits into
Open
Conversation
When an expression is replaced more than once in a module, maintain the replacement chain such that the intermediate replacements can all be cleaned up. Note that it is important that the expression name points at the original definition for the duration of the module. Fixes form-dev#882
Correctly handle products of vector wildcards: previously they were taken to be index wildcards, producing malformed term data. Fixes form-dev#665
These must check if expressions have been redefined, and follow the redefinition chain to the most recently-defined expression. Otherwise they act on stale redefined expressions, which are not processed in the module. Fixes form-dev#103
|
coverage: 65.769% (+0.02%) from 65.748% — jodavies:gpt-fix-experiments into form-dev:master |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These commits are from experimenting with using an LLM (OpenAI 5.6 Sol) to fix bugs in FORM. The general procedure was to add failing test cases to the test suite, and then prompt the LLM to find and fix the bug, including information about what the bug is and what my suspicions were about it.
This fixes #103, #665, #882. For "small" bugs like these, the resulting changes are compact and easy to reason about.
I have something for #607 and #608 but it is much more complicated and the scope quickly gets out of hand. This needs some discussions about the proper behaviour, I will open a discussion about that separately.
What are your thoughts on these, and LLM bug-fixing in general? I would say that with some focussed efforts, we could use LLMs to quickly fix many of the "easier" open bugs in the Issue tracker that no one had time to have a proper look at for years. This is good for FORM users. At the same time, by fixing things in this way we don't learn as much about the codebase, which is not a good thing in the long run.