Conversation
Each assertion type (Resolves, DoesNotResolve, ResolvesWith, ResolvesWithType, SearchByName, Needed) is now a self-contained class in src/babel_validation/assertions/, grouped by which service it targets (nodenorm.py, nameres.py, common.py). A central ASSERTION_HANDLERS registry in __init__.py maps lowercase assertion names to handler instances, and NodeNormAssertion / NameResAssertion marker base classes allow isinstance() checks for applicability. GitHubIssueTest.test_with_nodenorm() and test_with_nameres() are now 3-line dispatchers. A README.md documents all supported assertion types with examples in both wiki and YAML syntax. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…pendently. Bumps pytest to >=9.0 (which includes built-in subtests support). Each TestResult is now evaluated in its own subtest block, so a failure no longer short-circuits the rest. Adds post-loop state-consistency subtests: a closed issue with failing tests fails with a "consider reopening" message, and an open issue where all tests pass emits an xfail "consider closing" hint. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously, get_github_issue_id() and GitHubIssueTest.__str__() both resolved the org/repo name via github_issue.repository.organization.name, which triggers lazy PyGitHub API calls. Parse org/repo from html_url instead (always present in the issue JSON, no extra round-trip needed). Also resolves the TODO comment in get_test_issues_from_issue(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tests/pytest.ini takes precedence over pyproject.toml when running tests under tests/, so the [tool.pytest.ini_options] block was never read. Register the `unit` mark in tests/pytest.ini and remove the now-redundant section from pyproject.toml. Also fixes python-dotenv dependency name (was incorrectly listed as `dotenv`, a thin wrapper package). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove unreachable duplicate block in ResolvesWithTypeHandler.test_param_set - Use github_issue_id fixture param directly instead of re-parsing the URL - Replace unused *args with *_ in SearchByNameHandler Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If the cache file exists but contains invalid JSON (e.g. from a killed process mid-write), delete it and re-fetch rather than crashing with an unhelpful JSONDecodeError. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The full NameRes results dump belongs at DEBUG level, not on stdout, where it would bypass pytest's output capture and could produce large noise in test runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a GitHub-issue-driven testing framework to babel-validation by introducing a reusable src/babel_validation/ library (assertion handlers, service wrappers, and test-case sources) and wiring in unit-test CI.
Changes:
- Added an assertions framework (
src/babel_validation/assertions/) with auto-generated documentation and a unit test to keep it in sync. - Added GitHub Issues integration to discover/parse
{{BabelTest|...}}and YAMLbabel_tests:blocks and execute them against NodeNorm/NameRes. - Added caching/parallelism support (FileLock + xdist) and a GitHub Actions workflow to run unit tests on PRs.
Reviewed changes
Copilot reviewed 26 out of 36 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
pyproject.toml |
Adds new runtime/test dependencies for GitHub + caching + parallelism. |
uv.lock |
Locks new dependencies required by GitHub issue framework and xdist. |
src/babel_validation/core/testrow.py |
Introduces shared TestRow / TestResult types used across sources/assertions. |
src/babel_validation/services/nodenorm.py |
Adds cached wrapper for NodeNorm normalization calls. |
src/babel_validation/services/nameres.py |
Adds cached wrapper for NameRes lookup calls. |
src/babel_validation/sources/google_sheets/google_sheet_test_cases.py |
Moves Google Sheet loader into library + adds local CSV caching. |
src/babel_validation/sources/google_sheets/blocklist.py |
Extracts blocklist sheet loader into library module. |
src/babel_validation/sources/github/github_issues_test_cases.py |
Implements GitHub issue search + parsing of embedded BabelTest definitions. |
src/babel_validation/assertions/* |
Adds assertion handler base classes, NodeNorm/NameRes assertion implementations, and README generator. |
tests/github_issues/* |
Adds GitHub-issue-driven tests and fixtures, including caching of discovered issue IDs. |
tests/test_environment/test_assertions_docs.py |
Unit test to enforce auto-generated assertions README is current. |
tests/* (various) |
Updates imports to use new library modules; minor test data structure cleanup. |
tests/pytest.ini |
Adds unit marker + timeout configuration (but see comment about config discovery). |
.github/workflows/tests.yaml |
Adds PR CI job running unit tests via uv run pytest -m unit. |
.gitignore |
Ignores root .env. |
CLAUDE.md |
Updates architecture docs to reflect new src/babel_validation/ library and GitHub issues tests. |
Comments suppressed due to low confidence (1)
src/babel_validation/sources/google_sheets/google_sheet_test_cases.py:47
- The Google Sheet CSV download is cached to a temp file, but the HTTP response is never validated. If Google returns a non-200 (or an HTML error page), that content will be cached and can make subsequent runs fail in confusing ways. Call response.raise_for_status() (and ideally set a reasonable timeout) before persisting response.text to the cache file.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
tests/pytest.ini was never picked up when running pytest from the repo root (pytest searches upward, not into subdirectories). The timeout and unit marker registration were silently ignored. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The list of repositories to scan for BabelTest assertions is now configuration rather than code. It lives in [DEFAULT] so it applies globally and can be overridden per-section if ever needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
5c97356 to
552d53a
Compare
Add module docstring to github_issues_test_cases.py defining assertion, param_set, and param_sets; document GitHubIssueTest class and __init__ params; add inline comments showing how Wiki/YAML syntax maps to param_sets. Add param_sets/:param params: notes to base class methods in __init__.py. Add inline comments to HasLabelHandler, ResolvesWithTypeHandler, and SearchByNameHandler clarifying special element positions. Add "Param Sets" section to gen_docs.py INTRO and regenerate README.md. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both handlers previously silently discarded extra parameters. They now yield a failed TestResult when more than 2 params are supplied, making malformed assertions visible rather than silently tolerated. Added unit tests to cover the new behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers the unhandled path in get_test_issues_from_issue() where a YAML block matches the detection regex but contains invalid YAML. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…/null babel test content Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds TestIssueHasTests (5 tests) and TestGetIssuesWithTests (4 tests) covering the regex gate, deduplication logic, and search false-positive filtering in GitHubIssuesTestCases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove unused TestResult.github_issue_test field - Replace magic-number string slicing with removeprefix/removesuffix - Update stale TODO comment in CachedNameRes to describe actual limitation - Deduplicate CURIEs with set comprehension before bulk prewarm call - Rename github_issues_test_cases_fixture -> github_issues_test_cases Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR introduces a GitHub-issue-driven testing framework for babel-validation by adding a reusable src/babel_validation/ library layer (assertion handlers, GitHub issue parsing, service wrappers) and wiring unit-test CI plus xdist-friendly caching.
Changes:
- Added a new assertions framework (
src/babel_validation/assertions/) with auto-generated documentation and a unit test to keep docs in sync. - Added GitHub issue integration (
src/babel_validation/sources/github/) and corresponding tests/fixtures to discover and execute embedded{{BabelTest|...}}and YAML-defined assertions. - Restructured shared test logic into
src/babel_validation/and updated tests/CI/configuration (pytest config moved intopyproject.toml, added xdist/cache coordination).
Reviewed changes
Copilot reviewed 27 out of 37 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
uv.lock |
Locks new dependencies (PyGitHub, filelock, pytest-xdist, dotenv, tqdm, etc.) for the new framework. |
pyproject.toml |
Adds dependencies and moves pytest configuration to [tool.pytest.ini_options]. |
.github/workflows/tests.yaml |
Adds CI job to run unit tests on pull requests. |
.gitignore |
Ignores root .env file. |
CLAUDE.md |
Updates architecture notes to reflect new src/babel_validation/ library layout and GitHub-issue tests. |
tests/conftest.py |
Adds cache cleanup and a --issue option + fixture plumbing. |
tests/targets.ini |
Adds a configurable list of GitHub repositories to scan for embedded BabelTests. |
tests/pytest.ini |
Removes legacy pytest.ini config (now in pyproject.toml). |
tests/test_environment/test_env.py |
Updates imports to use the new library path under src/babel_validation/. |
tests/test_environment/test_assertions_docs.py |
New unit test enforcing that assertions/README.md stays in sync with generators. |
tests/nodenorm/test_nodenorm_from_gsheet.py |
Updates imports to use the new library path. |
tests/nodenorm/test_nodenorm_descriptions.py |
Adjusts test data containers (sets → lists) for parametrization. |
tests/nameres/test_nameres_from_gsheet.py |
Updates imports to use the new library path. |
tests/nameres/test_blocklist.py |
Refactors blocklist loading into src/babel_validation/sources/google_sheets/blocklist.py. |
tests/github_issues/conftest.py |
New GitHub-issues test fixtures with file-lock cache coordination. |
tests/github_issues/test_system.py |
New unit tests documenting parsing/execution edge cases for embedded issue assertions. |
tests/github_issues/test_github_issues.py |
New dynamic tests that execute parsed issue assertions against NodeNorm/NameRes. |
src/babel_validation/core/testrow.py |
Introduces shared core datatypes (TestRow, TestResult, TestStatus). |
src/babel_validation/services/nodenorm.py |
Adds cached NodeNorm client wrapper used by issue-driven tests. |
src/babel_validation/services/nameres.py |
Adds cached NameRes client wrapper used by issue-driven tests. |
src/babel_validation/sources/google_sheets/google_sheet_test_cases.py |
Adds FileLock-based temp caching for Google Sheets CSV fetch and migrates TestRow to core. |
src/babel_validation/sources/google_sheets/blocklist.py |
New module extracting the blocklist loader into the library. |
src/babel_validation/sources/github/github_issues_test_cases.py |
New GitHub issue parser/executor that finds wiki/YAML BabelTest blocks. |
src/babel_validation/assertions/* |
New assertion handlers + README generator and generated documentation. |
Comments suppressed due to low confidence (1)
src/babel_validation/sources/google_sheets/google_sheet_test_cases.py:47
- The Google Sheets download is cached to a temp file, but the initial
requests.get(csv_url)has no timeout and does not callraise_for_status(). A slow/hung request can stall the entire test run, and HTTP errors could be cached as if they were CSV. Use an explicit timeout and validate the response status before writing to the cache.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| "pygithub>=2.8.1", | ||
| "pytest>=9.0.2", | ||
| "pytest-timeout>=2.4.0", | ||
| "pytest-xdist[psutil]", |
There was a problem hiding this comment.
tests/github_issues/test_github_issues.py uses the subtests fixture, but the repo dependencies don’t include pytest-subtests, so this test will error at collection time (fixture not found). Add pytest-subtests to pyproject.toml dependencies (and lockfile), or refactor the test to avoid the subtests plugin.
| "pytest-xdist[psutil]", | |
| "pytest-xdist[psutil]", | |
| "pytest-subtests", |
| self.logger.debug(f"Called NodeNorm {self} with params {params}") | ||
| response = requests.post(self.nodenorm_url + "get_normalized_nodes", json=params) | ||
| response.raise_for_status() | ||
| result = response.json() |
There was a problem hiding this comment.
CachedNodeNorm.normalize_curies() makes a network call without a timeout. If the service is slow/unresponsive, tests can hang until the global pytest timeout. Pass an explicit timeout to requests.post (and consider making it configurable) to keep runs predictable.
| self.logger.debug(f"Called NameRes {self} with params {params}") | ||
| response = requests.post(self.nameres_url + "bulk-lookup", json=params) | ||
| response.raise_for_status() | ||
| result = response.json() | ||
|
|
||
| for query in queries_to_be_queried: | ||
| self.cache[query] = result.get(query, None) | ||
|
|
||
| for query in cached_queries: | ||
| result[query] = self.cache[query] | ||
|
|
||
| time_taken_sec = (time.time_ns() - time_started) / 1E9 | ||
| self.logger.info(f"Looked up {len(queries_to_be_queried)} queries {queries_to_be_queried} (with {len(cached_queries)} queries cached) with params {params} on {self} in {time_taken_sec:.3f}s") | ||
|
|
||
| return result | ||
|
|
||
| def lookup(self, query, **params): | ||
| if query in self.cache: | ||
| return self.cache[query] | ||
|
|
||
| params['string'] = query | ||
| self.logger.debug(f"Querying NameRes with params {params}") | ||
|
|
||
| response = requests.post(self.nameres_url + "lookup", params=params) | ||
| response.raise_for_status() | ||
| result = response.json() |
There was a problem hiding this comment.
CachedNameRes.bulk_lookup() / lookup() make network calls without a timeout. This can cause the GitHub-issue-driven tests to hang indefinitely on a bad endpoint. Add explicit request timeouts (and ideally a shared default) to both calls.
| issues = [] | ||
| for issue_id in issue_ids: | ||
| if m := re.match(r'^([^/]+)/([^#]+)#(\d+)$', issue_id): | ||
| # org/repo#N | ||
| issue = self.github.get_repo(f"{m.group(1)}/{m.group(2)}").get_issue(int(m.group(3))) | ||
| issues.append(issue) | ||
| elif m := re.match(r'^([^/#]+)#(\d+)$', issue_id): | ||
| # repo#N — find repo in configured list | ||
| repo_name, num = m.group(1), int(m.group(2)) | ||
| for full_repo in self.github_repositories: | ||
| if full_repo.split('/')[1] == repo_name: | ||
| issues.append(self.github.get_repo(full_repo).get_issue(num)) | ||
| break | ||
| elif m := re.match(r'^(\d+)$', issue_id): | ||
| # N — try all configured repos | ||
| num = int(m.group(1)) | ||
| for full_repo in self.github_repositories: | ||
| try: | ||
| issues.append(self.github.get_repo(full_repo).get_issue(num)) | ||
| except GithubException: | ||
| pass |
There was a problem hiding this comment.
get_issues_by_ids() silently drops IDs it can’t resolve (e.g., repo#N where repo isn’t in github_repositories, or N not present in any repo). Callers like tests/github_issues/conftest.py then index [0], which can raise a confusing IndexError. Consider raising a clear error (or returning a mapping of requested→found) when an ID can’t be resolved/fetched.
| issues = [] | |
| for issue_id in issue_ids: | |
| if m := re.match(r'^([^/]+)/([^#]+)#(\d+)$', issue_id): | |
| # org/repo#N | |
| issue = self.github.get_repo(f"{m.group(1)}/{m.group(2)}").get_issue(int(m.group(3))) | |
| issues.append(issue) | |
| elif m := re.match(r'^([^/#]+)#(\d+)$', issue_id): | |
| # repo#N — find repo in configured list | |
| repo_name, num = m.group(1), int(m.group(2)) | |
| for full_repo in self.github_repositories: | |
| if full_repo.split('/')[1] == repo_name: | |
| issues.append(self.github.get_repo(full_repo).get_issue(num)) | |
| break | |
| elif m := re.match(r'^(\d+)$', issue_id): | |
| # N — try all configured repos | |
| num = int(m.group(1)) | |
| for full_repo in self.github_repositories: | |
| try: | |
| issues.append(self.github.get_repo(full_repo).get_issue(num)) | |
| except GithubException: | |
| pass | |
| issues: list[Issue.Issue] = [] | |
| unresolved_ids: list[str] = [] | |
| for issue_id in issue_ids: | |
| resolved = False | |
| if m := re.match(r'^([^/]+)/([^#]+)#(\d+)$', issue_id): | |
| # org/repo#N | |
| issue = self.github.get_repo(f"{m.group(1)}/{m.group(2)}").get_issue(int(m.group(3))) | |
| issues.append(issue) | |
| resolved = True | |
| elif m := re.match(r'^([^/#]+)#(\d+)$', issue_id): | |
| # repo#N — find repo in configured list | |
| repo_name, num = m.group(1), int(m.group(2)) | |
| for full_repo in self.github_repositories: | |
| if full_repo.split('/')[1] == repo_name: | |
| issues.append(self.github.get_repo(full_repo).get_issue(num)) | |
| resolved = True | |
| break | |
| elif m := re.match(r'^(\d+)$', issue_id): | |
| # N — try all configured repos | |
| num = int(m.group(1)) | |
| found_in_any_repo = False | |
| for full_repo in self.github_repositories: | |
| try: | |
| issues.append(self.github.get_repo(full_repo).get_issue(num)) | |
| found_in_any_repo = True | |
| except GithubException: | |
| # Ignore missing issues in this repo; we'll check overall below. | |
| pass | |
| if found_in_any_repo: | |
| resolved = True | |
| if not resolved: | |
| # Either the ID format is unsupported, the repo wasn't in self.github_repositories, | |
| # or the issue number could not be found in any configured repository. | |
| unresolved_ids.append(issue_id) | |
| if unresolved_ids: | |
| msg = ( | |
| "Could not resolve the following GitHub issue IDs: " | |
| + ", ".join(sorted(set(unresolved_ids))) | |
| + ". Ensure they refer to valid issues in the configured repositories." | |
| ) | |
| self.logger.error(msg) | |
| raise ValueError(msg) |
| def _get_github_issues_test_cases() -> GitHubIssuesTestCases: | ||
| """Lazily construct GitHubIssuesTestCases, skipping tests if no token is available.""" | ||
| global _github_issues_test_cases | ||
| if _github_issues_test_cases is not None: | ||
| return _github_issues_test_cases | ||
| if not _github_token: | ||
| pytest.skip( | ||
| "GITHUB_TOKEN environment variable not set; skipping GitHub issues tests.", | ||
| allow_module_level=True, | ||
| ) | ||
| _github_issues_test_cases = GitHubIssuesTestCases(_github_token, _repos) | ||
| return _github_issues_test_cases |
There was a problem hiding this comment.
The github_issues_test_cases fixture hard-skips all GitHub issue tests when GITHUB_TOKEN is not set. This also skips parser-only unit tests (e.g., tests/github_issues/test_system.py) that don’t need live GitHub access. Consider allowing a “no-auth” mode for pure body-parsing (or constructing GitHubIssuesTestCases with a dummy client) so unit tests can still run in CI without requiring a token.
| try: | ||
| os.unlink(os.path.join(tempfile.gettempdir(), 'babel_validation_issues_cache.json')) | ||
| except FileNotFoundError: | ||
| pass |
There was a problem hiding this comment.
Cache cleanup only deletes babel_validation_gsheet_*.csv files but leaves their corresponding .lock files (and likewise leaves babel_validation_issues_cache.lock). Over time this can accumulate temp files. Consider deleting the lock files alongside the caches.
| try: | |
| os.unlink(os.path.join(tempfile.gettempdir(), 'babel_validation_issues_cache.json')) | |
| except FileNotFoundError: | |
| pass | |
| # Also remove any lock file associated with this cache file. | |
| lock_file = f + '.lock' | |
| try: | |
| os.unlink(lock_file) | |
| except FileNotFoundError: | |
| pass | |
| issues_cache = os.path.join(tempfile.gettempdir(), 'babel_validation_issues_cache.json') | |
| try: | |
| os.unlink(issues_cache) | |
| except FileNotFoundError: | |
| pass | |
| # Remove the lock file for the issues cache, if present. | |
| issues_cache_lock = issues_cache + '.lock' | |
| try: | |
| os.unlink(issues_cache_lock) | |
| except FileNotFoundError: | |
| pass |
This PR introduces a full GitHub-issue-driven test framework for babel-validation. Key additions: