Skip to content

Add GitHub issue tests#67

Open
gaurav wants to merge 87 commits intomainfrom
add-github-issue-tests
Open

Add GitHub issue tests#67
gaurav wants to merge 87 commits intomainfrom
add-github-issue-tests

Conversation

@gaurav
Copy link
Copy Markdown
Collaborator

@gaurav gaurav commented Jan 8, 2026

This PR introduces a full GitHub-issue-driven test framework for babel-validation. Key additions:

  • Assertion framework (src/babel_validation/assertions/) with 8 handler types
  • GitHub integration via PyGitHub — parses {{BabelTest|...}} wiki syntax and YAML blocks embedded in issue bodies
  • Restructure of shared code from tests/common/ into src/babel_validation/ (library pattern)
  • CI workflow running unit tests on PRs
  • pytest-xdist parallelism with FileLock-based cache coordination

gaurav and others added 10 commits February 15, 2026 02:39
Each assertion type (Resolves, DoesNotResolve, ResolvesWith, ResolvesWithType,
SearchByName, Needed) is now a self-contained class in
src/babel_validation/assertions/, grouped by which service it targets
(nodenorm.py, nameres.py, common.py). A central ASSERTION_HANDLERS registry
in __init__.py maps lowercase assertion names to handler instances, and
NodeNormAssertion / NameResAssertion marker base classes allow isinstance()
checks for applicability. GitHubIssueTest.test_with_nodenorm() and
test_with_nameres() are now 3-line dispatchers. A README.md documents all
supported assertion types with examples in both wiki and YAML syntax.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gaurav and others added 2 commits February 19, 2026 19:03
…pendently.

Bumps pytest to >=9.0 (which includes built-in subtests support). Each
TestResult is now evaluated in its own subtest block, so a failure no longer
short-circuits the rest. Adds post-loop state-consistency subtests: a closed
issue with failing tests fails with a "consider reopening" message, and an open
issue where all tests pass emits an xfail "consider closing" hint.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously, get_github_issue_id() and GitHubIssueTest.__str__() both resolved
the org/repo name via github_issue.repository.organization.name, which triggers
lazy PyGitHub API calls. Parse org/repo from html_url instead (always present
in the issue JSON, no extra round-trip needed). Also resolves the TODO comment
in get_test_issues_from_issue().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gaurav and others added 5 commits March 15, 2026 03:27
tests/pytest.ini takes precedence over pyproject.toml when running
tests under tests/, so the [tool.pytest.ini_options] block was never
read. Register the `unit` mark in tests/pytest.ini and remove the
now-redundant section from pyproject.toml.

Also fixes python-dotenv dependency name (was incorrectly listed as
`dotenv`, a thin wrapper package).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove unreachable duplicate block in ResolvesWithTypeHandler.test_param_set
- Use github_issue_id fixture param directly instead of re-parsing the URL
- Replace unused *args with *_ in SearchByNameHandler

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If the cache file exists but contains invalid JSON (e.g. from a killed
process mid-write), delete it and re-fetch rather than crashing with an
unhelpful JSONDecodeError.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The full NameRes results dump belongs at DEBUG level, not on stdout,
where it would bypass pytest's output capture and could produce large
noise in test runs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a GitHub-issue-driven testing framework to babel-validation by introducing a reusable src/babel_validation/ library (assertion handlers, service wrappers, and test-case sources) and wiring in unit-test CI.

Changes:

  • Added an assertions framework (src/babel_validation/assertions/) with auto-generated documentation and a unit test to keep it in sync.
  • Added GitHub Issues integration to discover/parse {{BabelTest|...}} and YAML babel_tests: blocks and execute them against NodeNorm/NameRes.
  • Added caching/parallelism support (FileLock + xdist) and a GitHub Actions workflow to run unit tests on PRs.

Reviewed changes

Copilot reviewed 26 out of 36 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
pyproject.toml Adds new runtime/test dependencies for GitHub + caching + parallelism.
uv.lock Locks new dependencies required by GitHub issue framework and xdist.
src/babel_validation/core/testrow.py Introduces shared TestRow / TestResult types used across sources/assertions.
src/babel_validation/services/nodenorm.py Adds cached wrapper for NodeNorm normalization calls.
src/babel_validation/services/nameres.py Adds cached wrapper for NameRes lookup calls.
src/babel_validation/sources/google_sheets/google_sheet_test_cases.py Moves Google Sheet loader into library + adds local CSV caching.
src/babel_validation/sources/google_sheets/blocklist.py Extracts blocklist sheet loader into library module.
src/babel_validation/sources/github/github_issues_test_cases.py Implements GitHub issue search + parsing of embedded BabelTest definitions.
src/babel_validation/assertions/* Adds assertion handler base classes, NodeNorm/NameRes assertion implementations, and README generator.
tests/github_issues/* Adds GitHub-issue-driven tests and fixtures, including caching of discovered issue IDs.
tests/test_environment/test_assertions_docs.py Unit test to enforce auto-generated assertions README is current.
tests/* (various) Updates imports to use new library modules; minor test data structure cleanup.
tests/pytest.ini Adds unit marker + timeout configuration (but see comment about config discovery).
.github/workflows/tests.yaml Adds PR CI job running unit tests via uv run pytest -m unit.
.gitignore Ignores root .env.
CLAUDE.md Updates architecture docs to reflect new src/babel_validation/ library and GitHub issues tests.
Comments suppressed due to low confidence (1)

src/babel_validation/sources/google_sheets/google_sheet_test_cases.py:47

  • The Google Sheet CSV download is cached to a temp file, but the HTTP response is never validated. If Google returns a non-200 (or an HTML error page), that content will be cached and can make subsequent runs fail in confusing ways. Call response.raise_for_status() (and ideally set a reasonable timeout) before persisting response.text to the cache file.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Comment thread src/babel_validation/sources/google_sheets/blocklist.py Outdated
Comment thread tests/github_issues/test_github_issues.py Outdated
Comment thread pyproject.toml
Comment thread tests/conftest.py Outdated
Comment thread tests/pytest.ini Outdated
gaurav and others added 4 commits March 15, 2026 22:49
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
tests/pytest.ini was never picked up when running pytest from the repo
root (pytest searches upward, not into subdirectories). The timeout
and unit marker registration were silently ignored.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The list of repositories to scan for BabelTest assertions is now
configuration rather than code. It lives in [DEFAULT] so it applies
globally and can be overridden per-section if ever needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gaurav gaurav force-pushed the add-github-issue-tests branch from 5c97356 to 552d53a Compare March 16, 2026 04:54
gaurav and others added 8 commits March 16, 2026 00:54
Add module docstring to github_issues_test_cases.py defining assertion,
param_set, and param_sets; document GitHubIssueTest class and __init__
params; add inline comments showing how Wiki/YAML syntax maps to
param_sets. Add param_sets/:param params: notes to base class methods in
__init__.py. Add inline comments to HasLabelHandler,
ResolvesWithTypeHandler, and SearchByNameHandler clarifying special
element positions. Add "Param Sets" section to gen_docs.py INTRO and
regenerate README.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both handlers previously silently discarded extra parameters. They now
yield a failed TestResult when more than 2 params are supplied, making
malformed assertions visible rather than silently tolerated. Added unit
tests to cover the new behaviour.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers the unhandled path in get_test_issues_from_issue() where a YAML
block matches the detection regex but contains invalid YAML.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…/null babel test content

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds TestIssueHasTests (5 tests) and TestGetIssuesWithTests (4 tests)
covering the regex gate, deduplication logic, and search false-positive
filtering in GitHubIssuesTestCases.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove unused TestResult.github_issue_test field
- Replace magic-number string slicing with removeprefix/removesuffix
- Update stale TODO comment in CachedNameRes to describe actual limitation
- Deduplicate CURIEs with set comprehension before bulk prewarm call
- Rename github_issues_test_cases_fixture -> github_issues_test_cases

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a GitHub-issue-driven testing framework for babel-validation by adding a reusable src/babel_validation/ library layer (assertion handlers, GitHub issue parsing, service wrappers) and wiring unit-test CI plus xdist-friendly caching.

Changes:

  • Added a new assertions framework (src/babel_validation/assertions/) with auto-generated documentation and a unit test to keep docs in sync.
  • Added GitHub issue integration (src/babel_validation/sources/github/) and corresponding tests/fixtures to discover and execute embedded {{BabelTest|...}} and YAML-defined assertions.
  • Restructured shared test logic into src/babel_validation/ and updated tests/CI/configuration (pytest config moved into pyproject.toml, added xdist/cache coordination).

Reviewed changes

Copilot reviewed 27 out of 37 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
uv.lock Locks new dependencies (PyGitHub, filelock, pytest-xdist, dotenv, tqdm, etc.) for the new framework.
pyproject.toml Adds dependencies and moves pytest configuration to [tool.pytest.ini_options].
.github/workflows/tests.yaml Adds CI job to run unit tests on pull requests.
.gitignore Ignores root .env file.
CLAUDE.md Updates architecture notes to reflect new src/babel_validation/ library layout and GitHub-issue tests.
tests/conftest.py Adds cache cleanup and a --issue option + fixture plumbing.
tests/targets.ini Adds a configurable list of GitHub repositories to scan for embedded BabelTests.
tests/pytest.ini Removes legacy pytest.ini config (now in pyproject.toml).
tests/test_environment/test_env.py Updates imports to use the new library path under src/babel_validation/.
tests/test_environment/test_assertions_docs.py New unit test enforcing that assertions/README.md stays in sync with generators.
tests/nodenorm/test_nodenorm_from_gsheet.py Updates imports to use the new library path.
tests/nodenorm/test_nodenorm_descriptions.py Adjusts test data containers (sets → lists) for parametrization.
tests/nameres/test_nameres_from_gsheet.py Updates imports to use the new library path.
tests/nameres/test_blocklist.py Refactors blocklist loading into src/babel_validation/sources/google_sheets/blocklist.py.
tests/github_issues/conftest.py New GitHub-issues test fixtures with file-lock cache coordination.
tests/github_issues/test_system.py New unit tests documenting parsing/execution edge cases for embedded issue assertions.
tests/github_issues/test_github_issues.py New dynamic tests that execute parsed issue assertions against NodeNorm/NameRes.
src/babel_validation/core/testrow.py Introduces shared core datatypes (TestRow, TestResult, TestStatus).
src/babel_validation/services/nodenorm.py Adds cached NodeNorm client wrapper used by issue-driven tests.
src/babel_validation/services/nameres.py Adds cached NameRes client wrapper used by issue-driven tests.
src/babel_validation/sources/google_sheets/google_sheet_test_cases.py Adds FileLock-based temp caching for Google Sheets CSV fetch and migrates TestRow to core.
src/babel_validation/sources/google_sheets/blocklist.py New module extracting the blocklist loader into the library.
src/babel_validation/sources/github/github_issues_test_cases.py New GitHub issue parser/executor that finds wiki/YAML BabelTest blocks.
src/babel_validation/assertions/* New assertion handlers + README generator and generated documentation.
Comments suppressed due to low confidence (1)

src/babel_validation/sources/google_sheets/google_sheet_test_cases.py:47

  • The Google Sheets download is cached to a temp file, but the initial requests.get(csv_url) has no timeout and does not call raise_for_status(). A slow/hung request can stall the entire test run, and HTTP errors could be cached as if they were CSV. Use an explicit timeout and validate the response status before writing to the cache.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pyproject.toml
"pygithub>=2.8.1",
"pytest>=9.0.2",
"pytest-timeout>=2.4.0",
"pytest-xdist[psutil]",
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests/github_issues/test_github_issues.py uses the subtests fixture, but the repo dependencies don’t include pytest-subtests, so this test will error at collection time (fixture not found). Add pytest-subtests to pyproject.toml dependencies (and lockfile), or refactor the test to avoid the subtests plugin.

Suggested change
"pytest-xdist[psutil]",
"pytest-xdist[psutil]",
"pytest-subtests",

Copilot uses AI. Check for mistakes.
Comment on lines +40 to +43
self.logger.debug(f"Called NodeNorm {self} with params {params}")
response = requests.post(self.nodenorm_url + "get_normalized_nodes", json=params)
response.raise_for_status()
result = response.json()
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CachedNodeNorm.normalize_curies() makes a network call without a timeout. If the service is slow/unresponsive, tests can hang until the global pytest timeout. Pass an explicit timeout to requests.post (and consider making it configurable) to keep runs predictable.

Copilot uses AI. Check for mistakes.
Comment on lines +41 to +66
self.logger.debug(f"Called NameRes {self} with params {params}")
response = requests.post(self.nameres_url + "bulk-lookup", json=params)
response.raise_for_status()
result = response.json()

for query in queries_to_be_queried:
self.cache[query] = result.get(query, None)

for query in cached_queries:
result[query] = self.cache[query]

time_taken_sec = (time.time_ns() - time_started) / 1E9
self.logger.info(f"Looked up {len(queries_to_be_queried)} queries {queries_to_be_queried} (with {len(cached_queries)} queries cached) with params {params} on {self} in {time_taken_sec:.3f}s")

return result

def lookup(self, query, **params):
if query in self.cache:
return self.cache[query]

params['string'] = query
self.logger.debug(f"Querying NameRes with params {params}")

response = requests.post(self.nameres_url + "lookup", params=params)
response.raise_for_status()
result = response.json()
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CachedNameRes.bulk_lookup() / lookup() make network calls without a timeout. This can cause the GitHub-issue-driven tests to hang indefinitely on a bad endpoint. Add explicit request timeouts (and ideally a shared default) to both calls.

Copilot uses AI. Check for mistakes.
Comment on lines +203 to +223
issues = []
for issue_id in issue_ids:
if m := re.match(r'^([^/]+)/([^#]+)#(\d+)$', issue_id):
# org/repo#N
issue = self.github.get_repo(f"{m.group(1)}/{m.group(2)}").get_issue(int(m.group(3)))
issues.append(issue)
elif m := re.match(r'^([^/#]+)#(\d+)$', issue_id):
# repo#N — find repo in configured list
repo_name, num = m.group(1), int(m.group(2))
for full_repo in self.github_repositories:
if full_repo.split('/')[1] == repo_name:
issues.append(self.github.get_repo(full_repo).get_issue(num))
break
elif m := re.match(r'^(\d+)$', issue_id):
# N — try all configured repos
num = int(m.group(1))
for full_repo in self.github_repositories:
try:
issues.append(self.github.get_repo(full_repo).get_issue(num))
except GithubException:
pass
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_issues_by_ids() silently drops IDs it can’t resolve (e.g., repo#N where repo isn’t in github_repositories, or N not present in any repo). Callers like tests/github_issues/conftest.py then index [0], which can raise a confusing IndexError. Consider raising a clear error (or returning a mapping of requested→found) when an ID can’t be resolved/fetched.

Suggested change
issues = []
for issue_id in issue_ids:
if m := re.match(r'^([^/]+)/([^#]+)#(\d+)$', issue_id):
# org/repo#N
issue = self.github.get_repo(f"{m.group(1)}/{m.group(2)}").get_issue(int(m.group(3)))
issues.append(issue)
elif m := re.match(r'^([^/#]+)#(\d+)$', issue_id):
# repo#N — find repo in configured list
repo_name, num = m.group(1), int(m.group(2))
for full_repo in self.github_repositories:
if full_repo.split('/')[1] == repo_name:
issues.append(self.github.get_repo(full_repo).get_issue(num))
break
elif m := re.match(r'^(\d+)$', issue_id):
# N — try all configured repos
num = int(m.group(1))
for full_repo in self.github_repositories:
try:
issues.append(self.github.get_repo(full_repo).get_issue(num))
except GithubException:
pass
issues: list[Issue.Issue] = []
unresolved_ids: list[str] = []
for issue_id in issue_ids:
resolved = False
if m := re.match(r'^([^/]+)/([^#]+)#(\d+)$', issue_id):
# org/repo#N
issue = self.github.get_repo(f"{m.group(1)}/{m.group(2)}").get_issue(int(m.group(3)))
issues.append(issue)
resolved = True
elif m := re.match(r'^([^/#]+)#(\d+)$', issue_id):
# repo#N — find repo in configured list
repo_name, num = m.group(1), int(m.group(2))
for full_repo in self.github_repositories:
if full_repo.split('/')[1] == repo_name:
issues.append(self.github.get_repo(full_repo).get_issue(num))
resolved = True
break
elif m := re.match(r'^(\d+)$', issue_id):
# N — try all configured repos
num = int(m.group(1))
found_in_any_repo = False
for full_repo in self.github_repositories:
try:
issues.append(self.github.get_repo(full_repo).get_issue(num))
found_in_any_repo = True
except GithubException:
# Ignore missing issues in this repo; we'll check overall below.
pass
if found_in_any_repo:
resolved = True
if not resolved:
# Either the ID format is unsupported, the repo wasn't in self.github_repositories,
# or the issue number could not be found in any configured repository.
unresolved_ids.append(issue_id)
if unresolved_ids:
msg = (
"Could not resolve the following GitHub issue IDs: "
+ ", ".join(sorted(set(unresolved_ids)))
+ ". Ensure they refer to valid issues in the configured repositories."
)
self.logger.error(msg)
raise ValueError(msg)

Copilot uses AI. Check for mistakes.
Comment on lines +33 to +44
def _get_github_issues_test_cases() -> GitHubIssuesTestCases:
"""Lazily construct GitHubIssuesTestCases, skipping tests if no token is available."""
global _github_issues_test_cases
if _github_issues_test_cases is not None:
return _github_issues_test_cases
if not _github_token:
pytest.skip(
"GITHUB_TOKEN environment variable not set; skipping GitHub issues tests.",
allow_module_level=True,
)
_github_issues_test_cases = GitHubIssuesTestCases(_github_token, _repos)
return _github_issues_test_cases
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The github_issues_test_cases fixture hard-skips all GitHub issue tests when GITHUB_TOKEN is not set. This also skips parser-only unit tests (e.g., tests/github_issues/test_system.py) that don’t need live GitHub access. Consider allowing a “no-auth” mode for pure body-parsing (or constructing GitHubIssuesTestCases with a dummy client) so unit tests can still run in CI without requiring a token.

Copilot uses AI. Check for mistakes.
Comment thread tests/conftest.py
Comment on lines +41 to +44
try:
os.unlink(os.path.join(tempfile.gettempdir(), 'babel_validation_issues_cache.json'))
except FileNotFoundError:
pass
Copy link

Copilot AI Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cache cleanup only deletes babel_validation_gsheet_*.csv files but leaves their corresponding .lock files (and likewise leaves babel_validation_issues_cache.lock). Over time this can accumulate temp files. Consider deleting the lock files alongside the caches.

Suggested change
try:
os.unlink(os.path.join(tempfile.gettempdir(), 'babel_validation_issues_cache.json'))
except FileNotFoundError:
pass
# Also remove any lock file associated with this cache file.
lock_file = f + '.lock'
try:
os.unlink(lock_file)
except FileNotFoundError:
pass
issues_cache = os.path.join(tempfile.gettempdir(), 'babel_validation_issues_cache.json')
try:
os.unlink(issues_cache)
except FileNotFoundError:
pass
# Remove the lock file for the issues cache, if present.
issues_cache_lock = issues_cache + '.lock'
try:
os.unlink(issues_cache_lock)
except FileNotFoundError:
pass

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants