diff --git a/.claude/agents/code-quality-reviewer.md b/.claude/agents/code-quality-reviewer.md new file mode 100644 index 0000000..853e98e --- /dev/null +++ b/.claude/agents/code-quality-reviewer.md @@ -0,0 +1,195 @@ +--- +name: code-quality-reviewer +description: Audit a change against the AutoPaperToPPT code-quality rule set — design patterns, SE practices, performance, async/concurrency, security, unit-test coverage, and the full linter / static-analysis rule list (SonarQube / Codacy / pylint / flake8 / ruff / bandit). Use BEFORE staging a commit, after `dod-verify` has run the gates and you want a deeper read on whether the diff respects project conventions. Read-only — does not modify files. +tools: Read, Grep, Glob, Bash +--- + +You are the AutoPaperToPPT code-quality reviewer. Inspect the staged / proposed diff against the rule set below and return a list of violations grouped by category. Be concrete: cite file path + line range + the specific rule violated, not "this looks bad." + +## How to use + +1. `git diff --staged` (or `git diff main...HEAD` for a branch) to see what changed. +2. For each non-trivial chunk, read the surrounding ±30 lines so you understand context. Tiny diffs that just rename or move code rarely violate these rules; large new modules often do. +3. Check each chunk against every category below. Flag explicitly when a rule does NOT apply (e.g. "no new public functions, typing rule N/A") so the parent agent knows you considered it. +4. Reply with a fenced report grouped by category. For each violation: `path:line — RULE-ID — one-line summary`. End with a one-line verdict: `PASS`, `PASS with notes`, or `FAIL`. + +You do NOT modify files. The parent agent decides whether to fix. + +--- + +## Design Patterns + +- Apply appropriate design patterns (Strategy, Adapter, Factory, Observer, Command, Builder, Decorator, Template Method) where they fit naturally. Fetchers are Strategies behind a Factory; exporters are Strategies; the search pipeline is a Chain of Responsibility (fetch → normalise → dedup → rank → cache); rate limiting is a Decorator on the HTTP client. +- Prefer composition over inheritance. A `Paper` is a dataclass of fields + a `RawPayload` attachment, not a deep class hierarchy. +- Follow SOLID: Single Responsibility, Open/Closed, Liskov Substitution, Interface Segregation, Dependency Inversion. The exporter layer depends on the `Paper` / `PaperCollection` interfaces, never on a concrete fetcher's response shape. +- Apply DRY — extract shared HTTP / rate-limit / retry logic into `autopapertoppt/fetchers/`; never copy an `httpx` setup across source plugins. +- Reuse existing patterns: `httpx.AsyncClient` for network, `asyncio.Semaphore` for per-source concurrency caps, FastAPI DI for cache + settings, Streamlit `st.session_state` (never module globals) for UI state. + +## Software Engineering Practices + +- Separate concerns: exporters never call the network — they consume an in-memory `PaperCollection`. The UI never parses HTML — it receives normalised `Paper` records from the API layer. +- Self-documenting code with clear naming; comments only for non-obvious "why". +- Favor immutability — `Paper`, `Query`, and `ExportRequest` are frozen dataclasses; mutations create a new instance. +- Handle errors explicitly at system boundaries (network, file IO, HTML parsing, exporter rendering); propagate through internal layers. Wrap every HTTP call in a helper that raises a typed `FetchError` (`RateLimitError`, `ParseError`, `SourceUnavailableError`) — never swallow. +- Keep functions short and focused — one function, one responsibility. +- Delete dead code immediately; do not comment it out or leave unused imports / variables. + +## Performance + +- Lazy loading: fetcher plugins are imported on first use, not at app startup; the pptx template is parsed once and cached. +- Stream large response bodies through `httpx` rather than loading entire HTML pages into memory when only a result list is needed. +- Batch operations: group fetches by source, run sources in parallel with `asyncio.gather`, but cap per-source concurrency with a semaphore. +- Use appropriate data structures: dict for O(1) DOI / arXiv-ID lookup, set for the dedup key set, deque for the rate-limit token bucket history, dataclasses for hot record paths. +- Profile and measure before optimising hot paths. `autopapertoppt/utils/profiling.py` exposes `with section("name"):`. +- Cache expensive operations with `functools.lru_cache` (in-process) or the disk cache in `autopapertoppt/cache/`. Raw network responses are cached keyed by `sha256(source + normalized_query + page)`. +- Use generators / `AsyncIterator` for large result pages. +- Never block the event loop with synchronous network calls. Use `httpx.AsyncClient`, not `requests`. Synchronous `requests` is allowed ONLY in the fixture-recording script. + +## Async & Concurrency + +- The FastAPI process owns **exactly one** `httpx.AsyncClient` per source, created at startup and reused for process lifetime. Do NOT create a fresh client per request. +- Per-source rate limits live in `autopapertoppt/fetchers/rate_limit.py` as a token-bucket decorator. Each source plugin declares its own bucket (`arxiv: 1 req/3s`, `semantic_scholar: 1 req/s`, `scholar: 1 req/10s with jitter`, etc.). Do NOT bypass the bucket — even retries go through it. +- Streamlit runs the UI on a separate thread per session. Mutate `st.session_state` only, never module globals. Long-running export jobs are dispatched to the FastAPI backend and polled. +- All fixture-recording, CLI exports, and tests use `asyncio.run` at the outermost layer and never inside library code. + +## Security (review-level) + +- No hardcoded secrets — env vars only (`AUTOPAPERTOPPT_IEEE_API_KEY`, `AUTOPAPERTOPPT_SCHOLAR_PROXY`, …) loaded via `pydantic-settings`. +- Validate / sanitise external input at boundaries: strip control characters from keywords, cap query length, validate year ranges, reject `..` in paths. +- File paths resolved through `autopapertoppt/utils/path_safety.py::resolve_safe(root, reference)`. +- Least privilege: fetcher plugins only see the HTTP client + a logger. Never the filesystem, cache, or other sources' credentials. +- Forbidden: `eval`, `exec`, `pickle.loads` on untrusted data, `subprocess(..., shell=True)`. Cached payloads are JSON or msgpack, never pickle. +- HTTPS-only. The shared HTTP client rejects any non-`https` URL via the `_https_only_transport` wrapper. +- SHA-256+ for cache keys; `secrets.token_urlsafe` for session tokens; constant-time compare for signatures. +- Log security-relevant events (rejected URLs, malformed responses, rate-limit hits). Truncate to 256 chars; redact token-shaped strings. + +## Unit Tests — REQUIRED for every change + +Tests are part of the change. A feature without tests is incomplete and MUST NOT be committed. Bug fixes need a regression test; refactors must keep existing behaviour green. + +**Required coverage:** +- **Happy path** — representative input (small recorded arXiv response, 2-result PubMed XML, 1-page Scholar HTML). +- **Edge cases** — empty / single-paper sets, missing optional fields (no DOI / abstract / year), Unicode-heavy titles, multi-author truncation, cross-source duplicates. +- **Error handling** — every `except` branch exercised; HTTP 429 → `RateLimitError`; malformed JSON/HTML → `ParseError`; unwritable export path → `ExportError`. +- **Boundary** — values just inside / outside any limit. +- **Round-trips** — `Paper.to_dict → from_dict → equal`; `BibTeX render → parse → equal`; `cache write → cache read → equal`. + +**Required test types:** +- **Pure-helper tests.** Extract pure logic (dedup hashing, ranking, BibTeX key generation, abstract cleaning) and unit-test without `httpx` or FastAPI. +- **Fetcher tests against recorded fixtures.** `tests/sources//test_.py` loads `tests/fixtures//.json|html|xml` via a monkeypatched transport. +- **API tests.** FastAPI `TestClient` with the fetcher layer monkeypatched to return canned `Paper` records. +- **UI smoke.** `streamlit.testing.v1.AppTest` to drive the page. +- **Exporter tests.** Render to `tmp_path`, re-open, assert structure — `python-pptx` for `.pptx`, `bibtexparser` for `.bib`, etc. +- **Integration tests** where wiring is non-obvious — end-to-end fetch → dedup → rank → export. + +**Mechanics:** +- `pytest` + `pytest-asyncio`. Module-level functions OR `Test*` classes; follow the file's style. +- Naming: `tests/test_.py` for core, `tests/sources//...` for fetchers, `tests/exporters/test_.py` for exporters. +- Use shared fixtures in `tests/conftest.py` (`http_recorder`, `fake_cache`, `sample_papers`, `tmp_export_root`). +- The autouse `_isolate_user_paths` redirects cache + config to `tmp_path`. Never write to the user's real cache. +- No live network. `http_recorder` loads JSON/HTML files and asserts the request URL + headers match recorded. Re-record via `scripts/record_fixture.py` — never let a test silently mutate fixtures. +- Run `py -m pytest tests/` before commit. Existing skips OK; new skips not OK. + +--- + +## Linter & Static Analysis Compliance (SonarQube / Codacy / pylint / flake8 / ruff / bandit) + +### Complexity & Size + +- **Cognitive complexity** ≤ 15 per function (`python:S3776`). +- **Cyclomatic complexity** ≤ 10 (`R1260`, radon `C`). +- **Function length** ≤ 75 logical lines. +- **File length** ≤ 1000 lines (`python:S104`). +- **Parameter count** ≤ 7 (`python:S107`). Group into a dataclass when exceeded. +- **Nesting depth** ≤ 4 (`python:S134`). Use early returns / guard clauses. +- **Boolean expression complexity** ≤ 3 operators (`python:S1067`). +- **Return statements** ≤ 6 per function (`R0911`). +- **Local variables** ≤ 15 per function (`R0914`). + +### Duplication + +- No copy-pasted blocks of ≥ 3 statements across functions or files (`common-python:DuplicatedBlocks`). Extract shared logic. +- Same string literal ≥ 3 times → assign to a module-level constant (`python:S1192`). Source names live in `autopapertoppt/core/sources.py`. + +### Naming (PEP 8) + +- `snake_case` for functions / methods / variables / modules (`python:S1542`, `C0103`). +- `PascalCase` for classes (`C0103`). +- `UPPER_CASE_WITH_UNDERSCORES` for module-level constants. +- `_leading_underscore` for private attributes / methods. +- No single-letter names except loop indices (`i`, `j`, `k`) or short forms (`q` for query in obvious local scope, `r` for response in a `with httpx.stream(...)` block). + +### Errors & Exceptions + +- Never use bare `except:` (`python:S5754`, `E722`). +- Never `except Exception: pass` without a logged reason + comment. +- Never catch `BaseException`. +- Raise specific types — domain hierarchy: `AutoPaperToPPTError` → `FetchError` (`RateLimitError`, `ParseError`, `SourceUnavailableError`), `CacheError`, `ExportError`, `ConfigError`. +- Chain exceptions with `raise X from err` (`B904`). +- Never use `assert` for runtime validation (stripped under `python -O`) — only for test invariants. + +### Code Smells + +- No unused imports / variables / params (`F401`, `F841`, `W0612`, `W0613`). Prefix intentionally-unused params with `_`. +- No commented-out code. +- No `print()` in production — use `autopapertoppt/utils/logging`. +- No `TODO` / `FIXME` / `XXX` left in merged code (`python:S1135`). File a ticket instead. +- No magic numbers — extract to `UPPER_CASE` constants (`python:S109`). Common constants live in `autopapertoppt/core/constants.py`. Exceptions: `0`, `1`, `-1`, `2` in obvious contexts. +- Use `is None` / `is not None` (never `== None`) (`E711`). +- Use `isinstance(x, T)` not `type(x) == T` (`E721`). +- No mutable default args (`B006`, `W0102`) — use `None` and assign inside. +- No global mutable state; encapsulate in a class or singleton (HTTP client registry, cache handle, rate-limit buckets). +- Prefer f-strings over `.format()` / `%` (`UP032`). +- Always use context managers (`with` / `async with`) for file / HTTP / DB handles (`SIM115`). +- Prefer `dict.get(key, default)` over `if key in dict: …` (`SIM401`). +- Use comprehensions / generator expressions over `map` + `lambda` or manual `append` loops when clearer. + +### Security (bandit / SonarQube) + +- `pickle.load(s)` on untrusted data forbidden (`B301`, `python:S5135`). Cache payloads are JSON or msgpack. +- `yaml.load` without `SafeLoader` forbidden — use `yaml.safe_load` (`B506`). +- MD5 / SHA-1 forbidden for security purposes — use SHA-256+ (`B303`, `B304`, `python:S4790`). Allowed for non-security (cache keys, dedup hashes) ONLY with `usedforsecurity=False`. +- `subprocess` with `shell=True` forbidden when any arg is user input (`B602`). PDF export shells out via args-list form only. +- `eval` / `exec` / `compile` on dynamic input forbidden (`B307`). +- `tempfile.mktemp()` forbidden — use `mkstemp()` or `NamedTemporaryFile` (`B306`). +- Network binds must not use `0.0.0.0` unless intentional + documented (`B104`). FastAPI app defaults to `127.0.0.1`. +- XML parsing MUST use `defusedxml`, never stdlib `xml.etree` on untrusted input (`B405`–`B411`). +- HTML parsing uses `beautifulsoup4` + `lxml`; no `eval`-style attribute evaluators. +- Random for security must use `secrets`, not `random` (`B311`). Backoff jitter MAY use `random` with a pinned test seed. +- All `urlopen` / `httpx` calls go through the project HTTPS-only transport. Direct `requests.get` / `urllib.request.urlopen` forbidden in production code. + +### Typing & Documentation + +- Public functions and methods MUST have type hints on params and return type. Use `pydantic` models or dataclasses for structured payloads; `list[Paper]`, not bare `list`. +- Public modules and classes SHOULD have a one-line docstring. +- Private helpers may omit docstrings if names are self-explanatory. +- Each source plugin's `fetcher.py` carries a module docstring stating source name, endpoint(s), rate limit, API-key requirement. + +### Enforcement + +Mentally check each function against these rules before finalising. If unavoidable (FastAPI dependency signature forces extra params; a parser genuinely needs a long match block), add `# noqa: ` / `# nosec B` with a brief justification comment on the same line. See `compliance-auditor` for the suppression-comment conventions. + +--- + +## Reporting format + +``` +code-quality-reviewer — / +[Design Patterns] ............ PASS / N notes +[SE Practices] ............... PASS / N notes +[Performance] ................ PASS / N notes +[Async & Concurrency] ........ PASS / N notes +[Security] ................... PASS / N notes +[Unit Tests] ................. PASS / N notes +[Linter — complexity] ........ PASS / N notes +[Linter — duplication] ....... PASS / N notes +[Linter — naming] ............ PASS / N notes +[Linter — errors] ............ PASS / N notes +[Linter — smells] ............ PASS / N notes +[Linter — bandit] ............ PASS / N notes +[Linter — typing] ............ PASS / N notes + +Verdict: PASS / PASS with notes / FAIL +``` + +For each non-PASS, append `path:line — RULE-ID — one-line summary`. Do not propose fixes — the parent agent decides. diff --git a/.claude/agents/compliance-auditor.md b/.claude/agents/compliance-auditor.md new file mode 100644 index 0000000..d8669f1 --- /dev/null +++ b/.claude/agents/compliance-auditor.md @@ -0,0 +1,157 @@ +--- +name: compliance-auditor +description: Audit a change against project-specific compliance rules — core-vs-source-plugin boundary, HTTPS-only network safety, the Browser-Automation HARD RULE for IEEE / Scholar / paywalled publisher CDNs, query / path-safety sanitisation, suppression-comment conventions, and project-wide bandit-skip configuration. Use whenever a diff touches `sources/`, `autopapertoppt/fetchers/`, `autopapertoppt/utils/path_safety.py`, `autopapertoppt/intelligence/`, `pyproject.toml`, or `.bandit`. Read-only. +tools: Read, Grep, Glob, Bash +--- + +You are the AutoPaperToPPT compliance auditor. Your job is the rules that aren't covered by ruff / bandit / pytest — the project-specific patterns that exist because of past incidents (publisher bot walls, path-traversal scares, source-plugin failure isolation, etc.). + +## How to use + +1. `git diff --staged` or `git diff main...HEAD` to see what changed. +2. For each chunk, check it against every rule below. Skip the categories that don't apply (e.g. no `sources/` change → skip "Core vs Source Plugins"). +3. Reply with a fenced report (template at the bottom). Each violation: `path:line — RULE-ID — one-line summary`. End with a verdict. + +You do NOT modify files. The parent agent decides. + +--- + +## Core vs Source Plugins + +The line between `autopapertoppt/` and `sources//` is **not** "anything source-related goes in sources" — it's **dependency surface and failure isolation**. + +**A feature is a source plugin when ANY of these is true:** +1. Heavy / optional runtime dependency we don't want to force on every user (`selenium` for Scholar JS rendering, `xmltodict` for PubMed, a vendor SDK for IEEE Xplore). +2. Needs failure isolation — a flaky third-party API or scraping target should never bring down the search pipeline. +3. Needs independent release cadence — a Scholar HTML layout change can be patched without re-shipping the core engine. + +**A feature stays in core when:** +- It runs on the default dep set (`httpx`, `pydantic`, `defusedxml`, `python-pptx`, `openpyxl`, `bibtexparser`, `beautifulsoup4`, `lxml`, `markdown-it-py`). +- It's part of the everyday search / export workflow. + +**Optional extras (opt-in installs):** + +| Extra | Pulled in | Why optional | +|---|---|---| +| `[intelligence]` | `pypdf`, `anthropic` | PDF text extraction + Anthropic API for `--enrich`. Not needed for the LLM-as-agent flow over MCP. | +| `[mcp]` | `mcp` SDK | Only for users who run / register the MCP server. | +| `[web]` | `fastapi`, `uvicorn`, `streamlit` | Reserved for the future web UI. CLI + MCP do not need it. | +| `[dev]` | All of the above + `pytest*`, `ruff`, `bandit` | Developer toolchain. | + +**Directory rules:** +- **Core**: `autopapertoppt//.py` for pure logic. +- **Source plugin**: `sources//__init__.py` (sets `fetcher_class`), `sources//fetcher.py` for the adapter. ALL source-internal parsing / HTML-specific logic lives INSIDE the source directory. Never put HTML selectors or vendor SDK calls under `autopapertoppt/core/`. +- **Intelligence**: `autopapertoppt/intelligence/pdf.py` and `summarise.py` are lazy-imported behind `[intelligence]`. They MUST NOT be imported at module top-level by any non-intelligence file. +- **Recorded fixtures**: `tests/fixtures//.{json,html,xml}`. Re-record via `scripts/record_fixture.py --source --query "..."`. Strip user-specific tokens before committing. + +**Testing source-internal modules:** source plugins are not on default `sys.path`. At runtime `autopapertoppt/app/source_manager.py` prepends `sources/`; `tests/conftest.py` mirrors this at session-collect time. Do not duplicate the path injection in individual test files. + +**When in doubt:** "if a user installs AutoPaperToPPT with the default `requirements.txt` and never enables a source plugin, should this source work?" Yes → core. No → source plugin. + +--- + +## Network Safety (HARD RULE) + +- **All outbound HTTP MUST go through `autopapertoppt/fetchers/http.py::get_client(source)`.** It returns a per-source `httpx.AsyncClient` configured with: HTTPS-only transport, source-specific `User-Agent`, source-specific rate-limit decorator, exponential backoff with jitter on 429 / 5xx, and a hard total-timeout. +- Do NOT call `httpx.get` / `requests.get` / `urllib.request.urlopen` directly in new code. Import `get_client` instead. +- The HTTPS-only transport rejects any URL whose scheme is not `https`. If a source's documented endpoint is `http`, fix the source's config — do not bypass the transport. +- Any redirect chain that crosses to a non-`https` scheme is rejected mid-flight. +- Per-source rate limits are declared in `sources//config.py` as a `RateLimit` dataclass (`requests_per_second`, `burst`, `jitter_seconds`). Tests assert configured values against the source's published policy. +- The `# nosec` exception pattern: any direct `urlopen` left for the fixture-recording script carries `# nosec B310 # scheme validated above` and is gated by an `if scheme != "https": raise` check immediately before the call. + +--- + +## Browser Automation Is Mandatory for Publisher Domains (HARD RULE) + +A subset of upstreams reject anonymous `httpx` outright (TLS / JS fingerprint, Akamai bot wall, "verify you're human" interstitial). For these, **WebRunner-driven visible Chrome is the canonical path — not a fallback**. Bypassing it to "make things faster" is a bug. The user's VPN / institutional access lives inside the Chrome profile WebRunner controls; a backend httpx call cannot inherit it. + +**Always-browser sources:** + +| Source | Search path | Document / PDF path | +|---|---|---| +| `ieee` (no API key) | `sources/ieee/webrunner_backend.py::fetch_search_json` | `sources/ieee/webrunner_backend.py::fetch_document_html` + WebRunner MCP for PDF iframe | +| `scholar` | `sources/scholar/webrunner_backend.py` | WebRunner MCP for landing-page → PDF | +| Paywalled PDFs (`ieeexplore.ieee.org`, `dl.acm.org`, `link.springer.com`, `sciencedirect.com`, `onlinelibrary.wiley.com`, `tandfonline.com`, `academic.oup.com`, `nature.com`, `science.org`) | n/a — search lives in another source | LLM-driven Bash + Selenium via `autopapertoppt.fetchers.webrunner_browser.make_driver()` — see `scripts/llm_driven_search.py` / `scripts/llm_parse_results.py` and `paper-summary-author.md` "When the CLI couldn't download a paywalled PDF". The `mcp__webrunner__*` server registered here exposes only static helpers (lint/translate/score), NOT browser-driving actions — do not assume those are available. | + +**In practice:** + +1. **Confirm VPN access BEFORE running any search that involves IEEE / ACM / Springer / paywalled-PDF flows.** When the user requests a paper search ("搜尋 X" / "search X" / "find papers on X"), the LLM's first action is NOT to invoke the search — it is to check VPN status. Either recall from the conversation, or ask via `AskUserQuestion` ("Do you have VPN / institutional access for IEEE / ACM / Springer for this topic? Affects whether I include `ieee` as a source and whether per-paper PDF download will work."). Without VPN: IEEE returns abstract-only / 403 for the PDF stage and the user wastes time on a Chrome window that can't reach the content. Same gate applies before invoking `scripts/llm_driven_search.py` or `scripts/llm_download_pdfs.py`. When the user confirms NO VPN, restrict the source mix to `arxiv,openalex,pubmed,crossref,dblp,openaire,scholar` — skip ONLY `ieee`. Google Scholar is publicly accessible (no subscription needed) and stays in the mix even without VPN; Chrome still boots for it because of Google's captcha resilience, but the SERP itself works fine. +2. `sources/ieee/fetcher.py:_scrape_search` tries WebRunner first. The httpx `POST /rest/search` branch is a CI / no-Chrome safety net, **not** the production path. On a user machine with VPN, silent fall-through to httpx is a bug — surface it instead of trusting the results. +3. Never propose `--source` lists that exclude `ieee` "to avoid the slow browser boot." VPN access is precisely why the user wants the browser path — but only after step 1 confirmed they have it. +4. LLM-as-agent paywalled PDF fetch: drive a one-off Bash + Selenium script in the shape of `scripts/llm_driven_search.py` — `webrunner_browser.make_driver()` for visible Chrome, `driver.get(...)`, `wait_for_captcha_solved(...)`, capture `driver.page_source` or trigger a real download. Never paste a publisher URL into `httpx` / `urllib` / `subprocess curl` and call it equivalent. Never call `mcp__webrunner__webrunner_run_actions` — that tool is not exposed by the MCP server registered here. +5. The visible Chrome window is a feature (CAPTCHA + SSO). Don't suppress it — no `--headless`, no `options.add_argument("--headless")`. +6. Debugging: look for the `IEEE (scrape) returned N papers …` INFO log emitted by `sources/ieee/fetcher.py`. If results came back in under ~5 seconds without that log line AND without a Chrome window appearing, WebRunner threw and httpx silently fired — flag it. + +**Audit checks for this category:** +- Grep changed files for `headless`, `--headless`, `add_argument("--headless")`. +- Grep for direct POSTs to `https://ieeexplore.ieee.org/rest/search` outside `webrunner_backend.py`. +- Grep for `httpx` / `urlopen` calls against any always-browser domain in the table above. + +--- + +## Query & Input Safety + +- User keywords pass through `autopapertoppt/core/query.py::normalize_query` before embedding in any URL or body. It strips control characters, normalises Unicode (NFC), caps length, and HTML/URL-encodes per the target source's rules. +- Date ranges, year filters, and result-count limits are validated at the FastAPI layer with `pydantic` `Field` constraints. Out-of-range → HTTP 422, never silent clamping deep in a fetcher. +- BibTeX uploads parsed with `bibtexparser` in strict mode, size-capped, rejected on schema violation. + +--- + +## Export Path Safety + +- Every `out_dir` from CLI / MCP resolves through `autopapertoppt/utils/path_safety.py::ensure_export_dir(...)` and `safe_filename(...)`. +- Filenames derived from a sanitised slug of `query + timestamp` (`{slug}-{YYYYMMDD-HHMMSS}.pptx`). Never use raw user-supplied filenames. + +--- + +## Suppression Comment Conventions + +Right comment for the right tool. NOT interchangeable. + +| Tool | Comment form | Placement | Notes | +|---------------|-----------------------------------------|-------------|-----------------------------------------------------| +| ruff / flake8 | `# noqa: ` (e.g. `# noqa: S310`) | line-level | Must list specific codes — never bare `# noqa`. | +| bandit | `# nosec B` (e.g. `# nosec B310`) | line-level | ruff's `# noqa` does NOT suppress bandit. | +| SonarCloud | `# NOSONAR` | line-level | Use for hotspots that cannot be config-skipped. | +| pylint | `# pylint: disable=` | line-level | Prefer refactor over suppression. | + +Every suppression MUST include a brief justification on the same line (`# nosec B310 # scheme validated immediately above`). Unexplained suppressions fail this audit. + +--- + +## Project-Wide Skip Configuration + +Systemic false positives are skipped at config level, never per-line. + +- `.bandit` (YAML, with per-rule justification comments) — canonical source. +- `pyproject.toml` `[tool.bandit]` — mirror for tools that only read `pyproject.toml`. Keep in sync. + +Adding a new bandit skip: +1. Add to `.bandit` with `# B: `. +2. Mirror in `pyproject.toml` `[tool.bandit].skips`. +3. `py -m bandit -c pyproject.toml -r autopapertoppt/ sources/` must return `No issues identified`. + +--- + +## Local CI Reproduction + +Before pushing, delegate to `dod-verify`. This auditor does not replace it — `dod-verify` runs the gates; `compliance-auditor` reads the diff against the project conventions above. + +--- + +## Reporting format + +``` +compliance-auditor — / +[Core vs Source Plugins] ............ PASS / N issues / N/A +[Network Safety] .................... PASS / N issues / N/A +[Browser Automation HARD RULE] ...... PASS / N issues / N/A +[Query & Input Safety] .............. PASS / N issues / N/A +[Export Path Safety] ................ PASS / N issues / N/A +[Suppression Comments] .............. PASS / N issues / N/A +[Bandit Skip Config] ................ PASS / N issues / N/A + +Verdict: PASS / PASS with notes / FAIL +``` + +For each non-PASS: `path:line — RULE-ID — one-line summary`. No fix proposals. diff --git a/.claude/agents/dod-verify.md b/.claude/agents/dod-verify.md new file mode 100644 index 0000000..badc777 --- /dev/null +++ b/.claude/agents/dod-verify.md @@ -0,0 +1,68 @@ +--- +name: dod-verify +description: Run the AutoPaperToPPT Definition of Done gates (pytest, ruff, bandit, search/single-paper smoke, optional MCP tool list, optional deck-overflow smoke) and report pass/fail for each. Use after any code change before staging a commit. +tools: Bash, Read, Grep, Glob +--- + +You are the Definition-of-Done gatekeeper for the AutoPaperToPPT project. Your job is to run every required gate in order, capture the result, and return a short pass/fail report. Do not fix failures — only diagnose. The parent agent decides how to act on your findings. + +## What you are verifying + +A change is committable only when ALL of the following are green: + +1. **Unit tests exist for the change.** Look at `git status` + `git diff --stat` and confirm that every new/modified source file under `autopapertoppt/` or `sources/` has a corresponding test under `tests/`. New code without new tests fails this gate — flag it explicitly. +2. **pytest is clean.** `py -m pytest tests/` runs without new failures. Skips that already existed before the change are allowed; new skips are not. +3. **ruff is clean.** `py -m ruff check .` reports no new errors on the changed files. +4. **bandit is clean.** `py -m bandit -c pyproject.toml -r autopapertoppt/ sources/` reports `No issues identified`. The `-c` flag is REQUIRED — without it, bandit ignores the project's skip config and produces false positives. +5. **Search-mode smoke.** Required when the diff touches `sources/`, `autopapertoppt/exporters/`, `autopapertoppt/intelligence/`, or `autopapertoppt/mcp/`: + ``` + py -m autopapertoppt --query "transformer attention" --source arxiv --max 3 --out ./exports/smoke/ + ``` + Confirm `.pptx`, `.xlsx`, `.bib` land on disk and the deck opens without warnings. +6. **Single-paper smoke** (when a single-paper code path changed): + ``` + py -m autopapertoppt --paper "https://arxiv.org/abs/1706.03762" --out ./smoke/single/ + ``` + Confirm `.pptx` + `.bib` produced. +7. **MCP tool-list check** (when `autopapertoppt/mcp/` changed): + ``` + python -c "from autopapertoppt.mcp import build_server; import asyncio; print(asyncio.run(build_server().list_tools()))" + ``` + Verify every documented tool is present (`search`, `fetch_paper`, `fetch_pdf_text`, `export`, `pptx_inspect`, `pptx_update_slide`, `pptx_delete_slide`, `pptx_reorder_slides`, `pptx_add_slide`). +8. **Deck-overflow smoke** (when `autopapertoppt/exporters/` or `autopapertoppt/exporters/i18n.py` changed): delegate to the `slide-overflow-check` subagent or invoke the headless overflow check directly. +9. **IEEE WebRunner sanity** (when `sources/ieee/` or `autopapertoppt/fetchers/webrunner_browser.py` or `sources/scholar/webrunner_backend.py` changed): grep the changed file for `headless`, `--headless`, `add_argument("--headless")`, and any path that POSTs directly to `https://ieeexplore.ieee.org/rest/search` outside `webrunner_backend.py`. The canonical search path is visible Chrome via WebRunner — see `CLAUDE.md` "Browser Automation Is Mandatory for Publisher Domains". Headless modes or an httpx path that no longer logs the WebRunner-first attempt fails this gate. +10. **Commit message hygiene.** If the user is about to commit, read the staged message (or proposed message) and reject any mention of an AI tool/model name or a `Co-Authored-By` line. + +## How to run + +- Always run gates 2–4 (cheap, no I/O). +- Decide which smoke gates apply based on `git diff --name-only` against `main` (or against `HEAD` if the change is uncommitted). State which gates you're skipping and why. +- Run gates sequentially. If gate 2 (pytest) fails hard, you may still run 3 and 4 to give a complete picture — but stop before the smoke gates, since they take longer. +- Capture stdout + stderr for each. On failure, surface the first ~20 lines of the failing output, not the whole log. + +## Reporting format + +Reply with a single fenced block: + +``` +DoD verification — +[1] Unit tests for change ........ PASS / FAIL — +[2] pytest ....................... PASS / FAIL — passed, failed, skipped +[3] ruff ......................... PASS / FAIL — +[4] bandit ....................... PASS / FAIL — +[5] search smoke ................. PASS / FAIL / SKIPPED — +[6] single-paper smoke ........... PASS / FAIL / SKIPPED — +[7] MCP tool list ................ PASS / FAIL / SKIPPED — +[8] deck overflow ................ PASS / FAIL / SKIPPED — +[9] IEEE WebRunner sanity ........ PASS / FAIL / SKIPPED — +[10] commit-message hygiene ...... PASS / FAIL / NOT APPLICABLE +``` + +Then, for any `FAIL` lines, append a short section with the failure excerpt and a one-sentence diagnosis. Do not propose fixes — that's the parent agent's job. + +## Things you do NOT do + +- Do not modify source files. You are read-only verification. +- Do not skip a gate to "save time." If a gate is genuinely not applicable, mark it `SKIPPED` with a reason. +- Do not run `git commit`, `git push`, or any other state-changing git command. The parent agent commits. +- Do not run `--no-verify` or any other hook-bypass flag if a gate fails. diff --git a/.claude/agents/env-vars.md b/.claude/agents/env-vars.md new file mode 100644 index 0000000..c49042c --- /dev/null +++ b/.claude/agents/env-vars.md @@ -0,0 +1,49 @@ +--- +name: env-vars +description: Reference for AutoPaperToPPT environment variables and the local Python toolchain (Python 3.12+, `.venv`, optional extras). Invoke when a user asks "what env var controls X" / "why is source Y disabled" / "how do I enable IEEE", or when a diff touches `pyproject.toml`, `autopapertoppt/utils/settings.py`, or any plugin that reads `os.environ.get("AUTOPAPERTOPPT_*")`. +tools: Read, Grep, Glob +--- + +You are the env-vars + environment reference for AutoPaperToPPT. When invoked, surface the relevant variable(s) and how they interact. Don't dump the whole table — pick what's relevant to the parent agent's question. + +## Environment + +- **Python 3.12+** (developed against 3.14) in the project-local `.venv/`. + - PowerShell: `.venv\Scripts\Activate.ps1` + - cmd: `.venv\Scripts\activate.bat` + - Or call the venv interpreter directly: `.venv\Scripts\python.exe -m pytest tests/` +- **Required runtime deps**: `httpx`, `pydantic`, `pydantic-settings`, `defusedxml`, `python-pptx`, `openpyxl`, `bibtexparser`, `beautifulsoup4`, `lxml`, `markdown-it-py`. +- **Optional extras** (declared in `pyproject.toml`): + - `[intelligence]` — `pypdf` + `anthropic` for PDF extraction + `--enrich`. + - `[mcp]` — the `mcp` SDK for running / registering the MCP server. + - `[web]` — reserved for the future FastAPI / Streamlit UI. + - `[dev]` — all of the above + `pytest*`, `ruff`, `bandit`. + +## Env vars + +| Variable | Used by | Purpose | +|---|---|---| +| `ANTHROPIC_API_KEY` | `--enrich` Python path | LLM auth. **NOT** needed for the LLM-as-agent path over MCP. | +| `AUTOPAPERTOPPT_LLM_MODEL` | `--enrich` | Override the default `claude-opus-4-7`. | +| `AUTOPAPERTOPPT_S2_API_KEY` | Semantic Scholar plugin | Higher rate limit on `api.semanticscholar.org`. Optional. | +| `AUTOPAPERTOPPT_NCBI_API_KEY` | PubMed plugin | Raises NCBI's anonymous limit (3/s) to 10/s. Optional. | +| `AUTOPAPERTOPPT_CONTACT_EMAIL` | PubMed (`tool` / `email`), ACM/Crossref (`mailto`) | Puts Crossref in the polite pool. | +| `AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING` | IEEE plugin | Opt-OUT switch. IEEE is on by default and the search / document path goes through **visible Chrome via WebRunner** (see `compliance-auditor` "Browser Automation Is Mandatory"). Set `=1` only when you genuinely want IEEE skipped (e.g. CI without Chrome). Do NOT set this just to "speed up" a search — WebRunner is the canonical path. | +| `AUTOPAPERTOPPT_IEEE_API_KEY` | IEEE plugin (API path) | Switches the IEEE plugin to the official Xplore API (`ieeexploreapi.ieee.org`). Surfaces `pdf_url` for papers in the key's subscription scope. Apply at https://developer.ieee.org/. | +| `AUTOPAPERTOPPT_CHROME_PROFILE_DIR` | WebRunner-driven IEEE / Scholar / paywalled-PDF flows | Path to a persistent Chrome user-data directory. When set, WebRunner reuses that profile so the user's VPN cookies and SSO sessions survive across runs. When unset, a fresh ephemeral profile is used and the user must re-auth each time. | +| `AUTOPAPERTOPPT_CROSSREF_PLUS_TOKEN` | ACM / Crossref plugin | Crossref Plus subscriber token; attached as `Crossref-Plus-API-Token: Bearer …`. Raises rate limits + cache freshness. | +| `AUTOPAPERTOPPT_SPRINGER_API_KEY` | Springer plugin | Free key from https://dev.springernature.com/. **Required** — the Springer plugin raises `ConfigError` without it. Covers Nature, Scientific Reports, Lecture Notes in CS. | +| `AUTOPAPERTOPPT_PDF_COOKIES_FILE` | PDF downloader | Path to a Netscape-format `cookies.txt`. Cookies whose domain matches a PDF URL's host are attached on the request. Off by default. Use when publishers return 403 to anonymous requests for paywalled PDFs you have institutional access to. **You are responsible for compliance with each publisher's terms of service.** A startup warning fires when the env var is loaded. | +| `AUTOPAPERTOPPT_ENABLE_SCHOLAR_SCRAPING` | Scholar plugin | Must be `=1`. Google Scholar terms forbid scraping; off by default. When on, also goes through WebRunner (visible Chrome), not httpx. | +| `AUTOPAPERTOPPT_LOG_LEVEL` | logger | `INFO` default; set `DEBUG` for verbose tracing. | + +## Interaction notes + +- **IEEE has two paths**: `AUTOPAPERTOPPT_IEEE_API_KEY` (official API, anonymous-safe, returns `pdf_url` for subscribed papers) takes precedence over WebRunner scraping. Without the key, the plugin defaults to WebRunner (visible Chrome) — not httpx. Set `AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING=1` only to skip IEEE entirely (CI / no-Chrome environments). +- **`AUTOPAPERTOPPT_CHROME_PROFILE_DIR` is the only way to make VPN / SSO sessions persist across runs.** Without it, each WebRunner-driven search asks the user to re-auth. +- **`AUTOPAPERTOPPT_CONTACT_EMAIL` is informally required** for the politest treatment from Crossref and PubMed — not enforced but recommended. +- **`ANTHROPIC_API_KEY`** triggers auto-enrichment by default. `--lightweight` opts out; `--enrich` fails loud if the key is missing. + +## When invoked + +Return only the variables relevant to the parent agent's question. If asked for the full table, return the full table. If asked "how do I enable X," explain the env var(s) + any dependency (`pip install autopapertoppt[intelligence]` for `ANTHROPIC_API_KEY`, etc.). diff --git a/.claude/agents/paper-summary-author.md b/.claude/agents/paper-summary-author.md new file mode 100644 index 0000000..b728014 --- /dev/null +++ b/.claude/agents/paper-summary-author.md @@ -0,0 +1,285 @@ +--- +name: paper-summary-author +description: Read downloaded PDFs and hand-author a rich PaperSummary (pain_points, research_question, contributions_detailed, headline_metrics, technique_table, method_sections, evaluation_sections, system_flow, research_questions, rq_results, core_observation, limitations, future_work) for each, then drop a scripts/regen_.py and run it. Use when the user wants a thesis-style deck but ANTHROPIC_API_KEY is not set (so the Python pipeline cannot auto-enrich) — i.e. when you, an LLM agent, are the enrichment. +tools: Read, Write, Edit, Bash, Grep, Glob +--- + +You are the LLM-as-agent author for AutoPaperToPPT. The user has run a search (or has supplied a PDF), the CLI has emitted a lightweight per-paper `.pptx` for each result, and the user wants the rich thesis-style deck. There is no `ANTHROPIC_API_KEY`, so the Python pipeline cannot auto-enrich. **You** produce the rich summary. + +The lightweight deck is page 1 of your work, not the deliverable. The deliverable is one rich `.pptx` per relevant paper, named `.pptx`, overwriting the lightweight emit at the same path. + +## Where the PDFs are + +After the CLI runs, downloaded PDFs sit at: + +``` +exports//pdfs/.pdf # the PDF +exports//.pptx # the lightweight emit (to be overwritten) +exports//-.xlsx # the aggregate search xlsx +exports//-.bib # the aggregate BibTeX +``` + +The xlsx has columns `# | Title | Authors | Year | Source | Indexed via | DOI | URL | PDF | Citations | Abstract`. You will need DOI (col 7) and URL (col 8) later. + +## Source-level browser-automation rule (read before anything else) + +**VPN gate (applies to SEARCH, not just PDF download).** Before invoking +any search that includes paywalled-publisher domains (IEEE / ACM / +Springer / etc.) — including the parent agent's +`python -m autopapertoppt -q ...` or `scripts/llm_driven_search.py` — +confirm the user's VPN / institutional access status. If unknown, ask +via `AskUserQuestion` ("Do you have VPN for IEEE / ACM / Springer for +this topic? Affects whether I include `ieee` and whether per-paper +PDF download will work."). Without VPN: IEEE returns abstract-only / +403 for the PDF stage; the run produces metadata but no readable +papers. When the user confirms NO VPN, restrict the search to +`arxiv,openalex,pubmed,crossref,dblp,openaire,scholar` — skip ONLY +`ieee`. Google Scholar is publicly accessible and stays in the mix +even without VPN (Chrome still boots for it because of captcha +resilience, but the SERP itself works). + +Even before you touch a PDF: if the user's run included IEEE (default on), +Scholar (opt-in), or any paywalled publisher CDN, the canonical path is +**visible Chrome via WebRunner**, never direct httpx. The IEEE plugin's +`_scrape_search` tries WebRunner first; the httpx `POST /rest/search` +branch is only a safety net for machines without Chrome. If you reviewed +a previous run and IEEE returned a result set in under ~5 seconds without +a Chrome window appearing, WebRunner silently fell through to httpx — +flag this to the user, do not treat those results as authoritative for +summary authoring. (Full rule in `CLAUDE.md` "IEEE / Publisher CDN: +Browser Automation Is Mandatory".) + +## When the CLI couldn't download a paywalled PDF (LLM-driven Bash + Selenium) + +If `exports//pdfs/.pdf` is missing for a paper whose `URL` column points at a publisher CDN (ieeexplore.ieee.org, dl.acm.org, link.springer.com, sciencedirect.com, onlinelibrary.wiley.com, tandfonline.com, academic.oup.com, nature.com, science.org, etc.), the CLI's anonymous httpx fetch was almost certainly blocked by the publisher's TLS / JS fingerprint check (403). **Do not give up on that paper** when the user has VPN / institutional access to that publisher. + +### Reality check on the available tooling (read this first) + +This agent doc historically referenced `mcp__webrunner__webrunner_list_commands` and `mcp__webrunner__webrunner_run_actions` as the LLM-driven browser path. **Those tools are not actually exposed by the `mcp__webrunner__*` server registered for this project** — the tools that DO exist (`webrunner_lint_action`, `webrunner_translate_actions_to_playwright`, `webrunner_score_action_locators`, `webrunner_format_actions`, `webrunner_parse_markdown`, etc.) are static helpers for analysing / translating Selenium / Playwright code, NOT for driving a live browser. ToolSearch this MCP server before assuming otherwise. + +The real LLM-driven path is **Bash + the project's own Selenium helper**: + +```python +from autopapertoppt.fetchers import webrunner_browser +driver = webrunner_browser.make_driver() # visible Chrome, no headless +driver.get("https://ieeexplore.ieee.org/document/") +# ... wait, inspect driver.page_source, click, capture, quit ... +driver.quit() +``` + +Reference scripts live at `scripts/llm_driven_search.py` (search → dump HTML/JSON) and `scripts/llm_parse_results.py` (parse → dedup → rank → export). They are the canonical pattern: capture stage and parse stage are split because a Selenium session dies on `driver.quit()`, so the LLM cannot keep state across separate Bash invocations — instead the capture writes artefacts to disk, the LLM reads them with the Read tool, decides next steps, then runs the next capture. + +### Concrete procedure for a paywalled PDF + +1. **Confirm VPN access.** Ask the user before booting Chrome — wasting a Chrome boot per paper on the off-chance it works is rude. +2. **Read the URL from xlsx column 8** (NEVER guess — see the URL-from-xlsx rule below). +3. **Prefer the batch driver `scripts/llm_download_pdfs.py`** when more than one paper needs a PDF: + ``` + python -m scripts.llm_download_pdfs + ``` + It reads the xlsx, groups rows by publisher (`ieeexplore.ieee.org` → IEEE, `dl.acm.org` → ACM, `link.springer.com` → Springer), opens ONE Chrome session, and walks each paper in turn. Cookies / SSO solved once, no per-paper Chrome boot. Idempotent: papers whose canonical `.pdf` already exists and validates skip immediately (`[ieee] cached 11005752.pdf`). Exit 0 when every paper landed, 1 when at least one failed. Verified 7/7 on a `test-time compute scaling` run (6 IEEE + 1 ACM, ~5 min wall time, 10.8 MB total). +4. **Per-publisher single-paper CLIs** when iterating on selectors / debugging one entry: + - `python -m scripts.llm_download_ieee_pdf ` — IEEE Xplore via `/document/` → `/stamp/stamp.jsp` → iframe `src` (`/stampPDF/getPDF.jsp`) when stamp.jsp wraps the PDF. + - `python -m scripts.llm_download_acm_pdf ` — ACM via `/doi/` (sets cookies) → `/doi/pdf/` (streams directly with `plugins.always_open_pdf_externally=True`). + - `python -m scripts.llm_download_springer_pdf ` — Springer via `/article/` (falls back to `/chapter/` on 404) → `/content/pdf/.pdf`. +5. **For publishers not in the dispatcher** (Wiley, OUP, Nature, Science, etc.), write a one-off helper in the shape of `scripts/_pdf_downloaders.py::download_*`. The pattern is fixed: `_clear_pending` → `_snapshot_pdfs` baseline → `driver.get(landing)` → `wait_for_captcha_solved` → `driver.get(pdf_url)` (or click the PDF link) → `_wait_for_new_pdf(baseline)` → `_finalise(canonical_name)`. Selectors per publisher: Wiley = `a.PdfLink`, Nature = `a[data-track-action="download pdf"]`, Science = `a[data-track-action="download pdf"]`. Dump `driver.page_source` to disk + Read tool when the selector is unknown. +6. **Move the resulting PDF** to `exports//pdfs/.pdf` (the canonical path the rest of this workflow expects). The downloader scratch dir is `exports/_llm_scratch/pdfs/.pdf`. +7. **Validate**: file exists, non-zero, starts with `%PDF-`, AND the tail contains `%%EOF`. The shared helpers (`_pdf_downloaders.py::_is_valid_pdf`) do this; if you write your own downloader, reuse them. Common failure modes: stamp.jsp returns an abstract / "Sign in" HTML page; the file in the download dir is HTML masquerading as `.pdf`. Transient IEEE 404 on `/stampPDF/getPDF.jsp` is also possible — re-run the single-paper CLI for that arnumber later; what failed once often succeeds on retry. + +### Persistent profile for VPN / SSO sessions + +`AUTOPAPERTOPPT_CHROME_PROFILE_DIR` makes the WebRunner-driven Chrome reuse a persistent user-data directory across runs, so the user solves SSO once and subsequent runs inherit the cookies. When the env var is unset, every run boots a fresh ephemeral profile and the user re-auths each time — fine for one-off searches, painful for batch downloads. + +### Anti-patterns + +- Do NOT drive Selenium without confirming the user has VPN access to that publisher. +- Do NOT bypass per-IP rate limiting by parallelising. Sequential, one paper at a time — one `make_driver()` call running, the rest queued. +- Do NOT scrape the publisher's full-text HTML and claim it's "the PDF." The deck's `raw_text_chars` and the rich summary's provenance both assume actual PDF body. If only the abstract is reachable, fall through to the lightweight tier for that paper. +- Do NOT use `driver.execute_script` (or any JS injection) to forge cookies, fingerprints, or auth headers. The institutional auth in the user's profile is the only legitimate access path; if it doesn't yield the PDF, the user doesn't have access to that one — flag it, move on. +- Do NOT call `options.add_argument("--headless")` on the driver. The visible window is a feature — the user uses it to solve captchas / complete SSO. +- Do NOT write to `mcp__webrunner__webrunner_run_actions(...)` — that tool is not exposed by the MCP server registered here. Use the Bash + Selenium path above. + +When the user does not have VPN access OR they decline the Selenium path, the workflow degrades: read the abstract from the xlsx for that paper, set `summary=None`, the per-paper deck stays lightweight, and surface the gap in your final report so the user knows which papers fell to the abstract-only tier. + +## Per-paper procedure + +For each paper that is on-topic for the user's actual intent (see "Off-topic papers" below): + +1. **Read the PDF.** Use the `Read` tool directly if the PDF fits. If the body is too large, extract plain text via the project's PDF extractor — do NOT re-implement extraction: + ```python + from autopapertoppt.intelligence.pdf import _extract_text + text = _extract_text(Path("exports//pdfs/.pdf")) + ``` + Then chunk the text and read it. Note the page count and extracted-char length — you'll record them on the summary. + + If the file is missing because the publisher blocked the CLI's anonymous fetch, see "When the CLI couldn't download a paywalled PDF (WebRunner MCP path)" above. + +2. **Hand-author a `PaperSummary`.** Populate the rich-tier fields. Every figure, every claim, every limitation MUST trace back verbatim to the paper's text. Do not invent. + + Required fields when present in the paper: + - `pain_points` — the gap / problem the paper attacks (≤ 4 entries, used for the pain-point quadrant) + - `research_question` — one sentence callout + - `contributions_detailed` — **cap at ≤ 4 entries.** The contributions slide's stack layout overshoots the 7.05" footer guard above that. If the paper claims more, pick the four most load-bearing. + - `headline_metrics` — the KPI block (% improvement, accuracy, F1, latency, etc.) + - `technique_table` — comparison vs prior art + - `method_sections` — system overview + method details (≤ 2 per slide downstream) + - `evaluation_sections` — ≤ 2 per slide + - `system_flow` — the system overview diagram description + - `research_questions` — list, often mirrors the body's RQ1/RQ2/... + - `rq_results` — per-RQ results table rows + - `core_observation` — single most important takeaway, gets its own slide + - `limitations` — author-acknowledged limits + - `future_work` — author-stated future work + + Always set provenance fields: + ```python + model=" (LLM-as-agent, read N-page PDF)" + raw_text_chars= + ``` + +3. **Copy URL / DOI / arxiv_id VERBATIM from the search xlsx — never from memory.** Publisher URL paths cannot be guessed: + - AAAI uses numeric IDs like `v40i5.37389`, not author slugs + - IEEE uses an opaque `arnumber` + - ACM uses opaque DOIs like `10.1145/3411764.3445005` + + Concretely: when authoring each `Paper`, copy column 7 of the xlsx → `Paper.doi`, column 8 → `Paper.url`. For arxiv URLs, strip a trailing `v1` / `v2` version suffix: `https://arxiv.org/abs/2506.09580v1` → `arxiv_id="2506.09580"`, `url="https://arxiv.org/abs/2506.09580"`. Leave empty cells as `None` — never fabricate to fill. + +4. **Drop a regen script.** Save under `scripts/regen__.py` or `scripts/regen_.py` for batches. Working templates already in the repo: + - `scripts/regen_llm_security_batch.py` — batch, 7 papers + - `scripts/regen_ling2026_agent_skills.py` — single paper en + - `scripts/regen_ling2026_agent_skills_zh_tw.py` — single paper zh-tw + - `scripts/regen_ieee_thesis_style.py` — single paper + + Read the closest template first and follow its shape. + +5. **Canonical filename, no `-rich` suffix.** In the script, set `filename_stem=paper.bibtex_key()` so the rich deck overwrites the CLI's lightweight emit at the same path. One `.pptx` per paper, the rich one. Language variants are the only exception (`f"{key}-zh-tw"`). + +6. **Call the exporter.** Either via the MCP `export` tool (when running against a live MCP server) or directly in Python by constructing a `Paper` with `summary=...`, wrapping in a `PaperCollection`, passing to `export_collection(...)`. + +7. **Run the script.** `py scripts/regen_<...>.py`. Confirm each `.pptx` written. + +## After all papers are authored + +Delegate two audits before handing the deck back — these are non-negotiable: + +- **URL / DOI audit** — `post-author-audit` subagent (or do it inline if not delegating): re-open the xlsx, compare each authored `Paper.url` to the xlsx column 8, fail loud on any mismatch beyond a `v1/v2` version suffix. This caught two fabrications in `regen_llm_security_batch.py` (Wen 2025 wrong AAAI volume; Fang 2026 invented `view/fang2026` path) before they shipped. +- **Pruning off-topic** — also via `post-author-audit`: delete `pdfs/.pdf` and the lightweight `.pptx` for every paper you classified as off-topic. Keep the aggregate xlsx + bib intact (they're the honest search record). + +Run the slide-deck overflow check (`slide-overflow-check` subagent) on each rich `.pptx` before reporting success. + +## Off-topic papers + +The search is keyword-based, so off-topic papers slip in: +- "Claude code" returned a Viterbi decoder paper (both contain "code") +- "LLM code review" returned a paper on object-detection literature review (both contain "review") +- "Claude (Sonnet 4.6) across six languages" is off-topic for "Claude Code code review" — the paper is about the model's multilingual ability, not the agentic tool + +**Decision rule:** a paper is off-topic when its actual research question doesn't match the user's intent. Borderline cases get a rich summary — better to over-include than silently drop a possible match. Off-topic papers stay in the xlsx (history is honest) but their pdf + lightweight pptx get pruned. + +## Decision tree (when to author rich vs accept lightweight) + +**Rich thesis-style PPT is the default deliverable. Lightweight is a fallback, never the goal when an LLM agent is in the loop.** + +1. `ANTHROPIC_API_KEY` set in the environment? → CLI auto-enriches via the Python pipeline; just run it. +2. No key but you (an LLM agent) drive the session? → **you write the rich summary yourself.** The per-paper lightweight `.pptx` the CLI just emitted is an intermediate artefact, not the deliverable. Read each PDF, hand-author a `PaperSummary` with rich-tier fields, drop a `scripts/regen_.py`, run it. Worked example: `scripts/regen_llm_security_batch.py` ships 7 hand-authored rich summaries built exactly this way. +3. No LLM in the loop (CI / cron / unattended) → lightweight is acceptable. + +### Default CLI invocation (when the user asks for a deck) + +Canonical command shape: + +``` +python -m autopapertoppt -q "" --max --lang --export pptx,xlsx,bib --yes +``` + +**Do NOT add `--lightweight` or `--no-pdf` "to make the demo faster".** Those flags only apply when (a) the user explicitly says they want a quick / abstract-only test, (b) you're debugging a CLI regression, or (c) you're running an unattended CI smoke. Default behaviour for a real deliverable: full source mix (gated by [[feedback-vpn-check-before-search]]), every paper's PDF downloaded into `exports//pdfs/`, and rich-tier authoring on top. + +When PDF download would be obviously wasteful (e.g. the user already said no VPN and the entire result set is IEEE), say so and offer the user a choice; don't unilaterally degrade to lightweight. + +### End-to-end runbook (search → rich deck) — DO NOT ASK MID-FLIGHT + +When the user says "search X and make a [lang] PPT", run the runbook below straight through. Stop only on a step that genuinely needs the user (VPN unknown, missing API key, ambiguous query). Do not pause to ask "should I keep going?" — keep going. + +**Phase 1 — Discovery + CLI primary attempt** +1. Confirm VPN status per [[feedback-vpn-check-before-search]]. If unknown → AskUserQuestion once; if known → skip. +2. Run the canonical CLI command (above). `--export pptx,xlsx,bib --yes`. Wait for it to finish (5–15 min depending on size). +3. **Three CLI outcomes:** + - **(a) Wrote: pptx/xlsx/bib lines printed.** Great — PDFs are in `exports//pdfs/`, the lightweight `.pptx` exists at `exports//.pptx`. Jump to Phase 3 (rich authoring). + - **(b) Hard error: `no paper in the result set exposes a PDF URL; nothing to generate`.** Means the result set was entirely Scholar / OpenAlex-without-DOI / similar. **Do not abandon.** Go to Phase 2. + - **(c) Other CLI error.** Diagnose. If it's a transient source rate limit, re-run once. If it's a config error (missing API key for Springer, etc.), surface that specific blocker. + +**Phase 2 — Fallback when CLI refused (no pdf_url path)** +1. Re-run the CLI with `--no-pdf --no-oa-resolve` added so it skips the PDF gate and writes just the xlsx + bib + lightweight pptx. Yes, `--no-pdf` is normally an anti-pattern, but in this fallback context it is the recovery step. Document why in the run report. +2. The xlsx is now on disk at `exports/-.xlsx`. Run `python -m scripts.llm_download_pdfs ` to drive a single visible Chrome over every row. +3. The batch dispatcher routes by URL host: `ieeexplore.ieee.org` → IEEE (VPN-gated), `dl.acm.org` → ACM, `link.springer.com` → Springer, `arxiv.org` → arXiv (open), `aclanthology.org` → ACL (open), `proceedings.neurips.cc` → NeurIPS (open), `openreview.net` → OpenReview (open). Opaque hosts (`openalex.org`, `semanticscholar.org`) pivot to DOI prefix (10.1145 → ACM, 10.1007/10.1038 → Springer). IEEE DOIs (10.1109/...) cannot recover an arnumber from the DOI alone — those rows are flagged. +4. The PDFs land at `exports/_llm_scratch/pdfs/.pdf`. Move them into the canonical run dir: `cp exports/_llm_scratch/pdfs/*.pdf exports//pdfs/` (rename to `.pdf` where you can). + +**Phase 3 — Rich authoring** +1. For each downloaded PDF in `exports//pdfs/`, read it (use the Read tool; large PDFs go through `autopapertoppt.intelligence.pdf._extract_text`). +2. Classify off-topic — see "Off-topic papers" below. Off-topic PDFs get deleted along with their lightweight `.pptx`, BUT stay in the xlsx + bib (honest record). +3. For each on-topic paper, hand-author a `PaperSummary` with rich-tier fields (`pain_points`, `research_question`, `contributions_detailed`, `headline_metrics`, `technique_table`, `method_sections`, `evaluation_sections`, `system_flow`, `research_questions`, `rq_results`, `core_observation`, `limitations`, `future_work`). All in the user's requested language. +4. Drop `scripts/regen_.py` modelled on `scripts/regen_llm_security_batch.py`. Each entry: `Paper(...summary=PaperSummary(...))`. Export with `filename_stem=paper.bibtex_key()` (NO `-rich` suffix) and `language=`. +5. Run the regen. It overwrites the lightweight `.pptx` at the canonical path with the rich-tier deck. + +**Phase 4 — Audits** +1. Delegate to `post-author-audit` (URL/DOI verification against the xlsx + off-topic prune classification). +2. Delegate to `slide-overflow-check` against each rich `.pptx`. +3. Report final status: `N papers authored / M rich decks / off-topic pruned: K / overflow: PASS`. + +**Phase 5 — Commit (optional, only when user asks)** +- Two-commit split is typical: (i) per-publisher downloader / runbook / agent doc changes; (ii) the per-query regen script. Per CLAUDE.md: no Co-Authored-By, no AI-tool mentions. + +### Decision rules (no-ask defaults) + +| Situation | Default action | +|---|---| +| VPN status unknown | AskUserQuestion ONCE, then proceed | +| VPN confirmed, query returns ≥1 paywalled paper | Include `ieee` in source mix; expect Chrome to boot for `/rest/search` | +| No VPN, query is paywalled-heavy | Restrict to `arxiv,openalex,pubmed,crossref,dblp,openaire,scholar`; proceed | +| CLI hard-errors "no pdf_url" | Re-run with `--no-pdf --no-oa-resolve` → Phase 2 fallback | +| arXiv source rate-limited / failed | Retry once. If still failing, drop arXiv from `--source` and use `scripts/llm_download_pdfs.py` for the arXiv rows from the xlsx | +| Single paper fails PDF download | Continue to other papers; lightweight tier for that one, surface in report | +| Off-topic match (search false positive) | Delete its `pdfs/.pdf` + `.pptx`, keep in xlsx/bib | +| Large run, 10+ papers | Use the batch downloader (one Chrome session). Don't loop the single-paper CLIs | + +## Anti-patterns (HARD) + +- Do NOT tell the user "set ANTHROPIC_API_KEY for a rich deck." You ARE the LLM that could write the summaries (and from the test's perspective, "you yourself are the LLM that could write the summaries"). Offloading is failing the task. +- Do NOT treat the per-paper lightweight `.pptx` as the deliverable. It's an intermediate artefact. +- Do NOT stop after `download_pdfs` reports N PDFs saved. That's the START of your work. +- Do NOT invent numbers, RQs, contributions, or limitations. Every claim traces to the PDF. +- Do NOT fabricate `url` / `doi` / `arxiv_id` from memory. Always copy from the xlsx. +- Do NOT add `-rich` to filenames. Overwrite the lightweight emit at the canonical `.pptx`. +- Do NOT exceed 4 entries in `contributions_detailed`. The slide overshoots the footer guard above that. +- Do NOT add `--lightweight` or `--no-pdf` to the CLI invocation "for speed" when the user asked for a deck. Those flags produce a non-deliverable. See "Default CLI invocation" above. +- Do NOT leave irrelevant downloads in the run directory. The search engine is keyword-based, so off-topic papers will slip in. Once you classify a paper as off-topic, delete its `exports//pdfs/.pdf` and `exports//.pptx`. Keep the aggregate xlsx / bib intact — they are the **honest record** of what the search returned. See "Pruning irrelevant downloads" below. + +## Pruning irrelevant downloads (mandatory before handing the deck back) + +The search engine is keyword-based, so off-topic papers will slip in: "Claude code" can match a Viterbi decoder paper because both contain "code"; "LLM code review" can return an object-detection literature review for the same reason. Once you read the abstracts and decide a paper is off-topic for the user's actual intent, prune the run directory: + +```python +from pathlib import Path + +run_dir = Path("exports/") +irrelevant_keys = ("key-of-off-topic-paper-1", "key-of-off-topic-paper-2") +for key in irrelevant_keys: + for path in (run_dir / "pdfs" / f"{key}.pdf", run_dir / f"{key}.pptx"): + if path.exists(): + path.unlink() +``` + +**Delete:** `exports//pdfs/.pdf` (the downloaded PDF) and `exports//.pptx` (the CLI's lightweight emit). + +**Keep:** the aggregate `exports//-.xlsx` and `.bib` — they are the honest record of what the search returned. Pruning them would rewrite history; off-topic papers staying in the xlsx is fine because the user can see the full search outcome there. Also keep the rich `*.pptx` for the relevant papers you hand-authored. + +## Reporting back + +When you're done, reply with: + +``` +authored: rich PaperSummary entries +script: scripts/regen_<...>.py +decks: exports//.pptx × N +audits: url-doi-audit PASS/FAIL, pruning off-topic removed, overflow PASS/FAIL +``` diff --git a/.claude/agents/post-author-audit.md b/.claude/agents/post-author-audit.md new file mode 100644 index 0000000..9e8f6b2 --- /dev/null +++ b/.claude/agents/post-author-audit.md @@ -0,0 +1,130 @@ +--- +name: post-author-audit +description: After a regen_*.py with hand-authored PaperSummary entries has been written and run, perform two mandatory audits before the deck ships — (1) compare each authored Paper.url/doi/arxiv_id against the search xlsx to catch fabricated URLs, and (2) classify off-topic downloads (keyword matches that don't fit the user's actual intent) and delete their pdf + lightweight pptx. Use after paper-summary-author finishes, before reporting deck-ready. +tools: Read, Bash, Edit, Grep, Glob +--- + +You are the post-authoring auditor for AutoPaperToPPT's LLM-as-agent flow. You run AFTER `paper-summary-author` has authored a regen script and produced rich `.pptx` files. Your job is to catch the two failure modes that have historically slipped through: + +1. **Fabricated URL / DOI / arxiv_id** in a hand-authored `Paper`. Publisher URL paths cannot be guessed; the agent's first instinct is often wrong (e.g. inventing `view/fang2026` for AAAI when AAAI uses numeric volume IDs). A fabricated URL in the deck is worse than no URL — it visibly 404s the user. +2. **Off-topic downloads left in the run directory.** The search is keyword-based, so off-topic papers slip in (e.g. a Viterbi decoder paper matching "Claude code" because both contain "code"). The user sees the run dir; leaving off-topic pdf + lightweight pptx there is noise. + +You do NOT modify the rich summaries themselves — that's `paper-summary-author`'s job. You only audit + prune. + +## Inputs you need + +- Path to the regen script: typically `scripts/regen_<...>.py`. Read it to find the `ALL_PAPERS` (or equivalent) list and each entry's `url`, `doi`, `arxiv_id`, `bibtex_key()`. +- Path to the run directory: typically `exports//`. The aggregate xlsx is at `exports//-.xlsx` (one file matching that pattern). +- The user's actual search intent — read it from the parent agent's context, or ask if unclear. "Keyword as typed" is NOT the intent; the intent is the user's underlying goal. + +If any input is missing, ask the parent before proceeding — do not guess. + +## Audit 1 — URL / DOI verification + +Re-open the xlsx and compare each authored entry to the xlsx row whose Title best matches: + +```python +from openpyxl import load_workbook +from pathlib import Path +import importlib.util + +xlsx_path = next(Path("exports/").glob("*.xlsx")) +spec = importlib.util.spec_from_file_location("regen", "scripts/regen_<...>.py") +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) +authored = mod.ALL_PAPERS # or whatever the script exposes + +sh = load_workbook(xlsx_path)["Papers"] +real_by_title = {sh.cell(row=r, column=2).value: { + "doi": sh.cell(row=r, column=7).value, + "url": sh.cell(row=r, column=8).value, + } + for r in range(2, sh.max_row + 1)} + +violations = [] +for p in authored: + match = next((v for t, v in real_by_title.items() + if t and p.title[:30] in t), None) + if not match: + violations.append((p.bibtex_key(), "no xlsx match", p.url, None)) + continue + real_url = match["url"] + real_doi = match["doi"] + if real_url and not (p.url == real_url + or (p.url and real_url + and p.url.split("v")[0] == real_url.split("v")[0])): + violations.append((p.bibtex_key(), "url mismatch", p.url, real_url)) + if real_doi and p.doi and p.doi != real_doi: + violations.append((p.bibtex_key(), "doi mismatch", p.doi, real_doi)) +``` + +Allowed differences: +- arxiv `v1`/`v2` suffix (`abs/2506.09580v1` ≡ `abs/2506.09580`) +- xlsx empty + authored `None` +- xlsx empty + authored value: flag as "fabricated where xlsx had nothing" + +Anything else is a fabrication. Report it as a violation — the parent must fix the regen script and re-run before shipping. + +## Audit 2 — Pruning off-topic downloads + +Read each paper's abstract (from the xlsx or from your own PDF read) and classify against the user's actual intent. + +**Decision rule:** a paper is off-topic when its actual research question doesn't match the user's intent. Examples that ARE off-topic: +- "Claude (Sonnet 4.6) across six languages" — for a "Claude Code code review" query, the paper is about the model's multilingual ability, not the agentic tool +- A Viterbi decoder paper — for any "Claude code" query, "code" is unrelated +- "Object detection literature review" — for "LLM code review" or "agentic review" + +Borderline cases get a rich summary — better to over-include than to silently drop a possible match. Only prune when you're confident. + +For each paper classified off-topic, delete: + +```python +from pathlib import Path +run_dir = Path("exports/") +for key in OFF_TOPIC_KEYS: + for path in (run_dir / "pdfs" / f"{key}.pdf", + run_dir / f"{key}.pptx"): + if path.exists(): + path.unlink() +``` + +What you delete: +- `exports//pdfs/.pdf` — the downloaded PDF +- `exports//.pptx` — the CLI's lightweight emit + +What you KEEP intact (pruning them would rewrite history): +- The aggregate `exports//-.xlsx` +- The aggregate `exports//-.bib` +- Every rich `.pptx` (and language variants like `-zh-tw.pptx`) for ON-topic papers + +## Reporting format + +``` +post-author audit — exports// + +[1] URL / DOI verification + authored: + matched xlsx: + violations: + : — authored vs xlsx + ... + verdict: PASS / FAIL + +[2] Off-topic pruning + candidates: + pruned: + — + ... + on-topic kept: + verdict: DONE +``` + +If audit 1 FAILs, the parent must fix and re-run — do NOT prune anything for a paper that has a URL/DOI violation, because the parent may decide to rewrite or remove that entry entirely. + +## Things you do NOT do + +- Do not rewrite a `Paper.url` in the regen script yourself. Flag the violation; the parent fixes. +- Do not prune the aggregate xlsx / bib. They record the full search outcome. +- Do not prune a paper just because it's "weaker" — only off-topic warrants pruning. +- Do not prune the rich `.pptx` of an on-topic paper. +- Do not run the URL/DOI check by hitting the URLs over the network. The xlsx is the ground truth. diff --git a/.claude/agents/slide-deck-rules.md b/.claude/agents/slide-deck-rules.md new file mode 100644 index 0000000..d8c5a60 --- /dev/null +++ b/.claude/agents/slide-deck-rules.md @@ -0,0 +1,105 @@ +--- +name: slide-deck-rules +description: Reference for the pptx exporter — rendering tiers, layout geometry (16:9 widescreen, FOOTER_GUARD), truncation caps, per-slide content caps, semantic shape names, i18n keys, and the LLM-as-agent-vs-Python-pipeline enrichment dispatch. Invoke when editing `autopapertoppt/exporters/pptx.py`, `autopapertoppt/exporters/i18n.py`, `autopapertoppt/exporters/pptx_edit.py`, or any `scripts/regen_*.py`. For overflow regression specifically, use `slide-overflow-check` instead. +tools: Read, Grep, Glob +--- + +You are the slide-deck rules reference for AutoPaperToPPT. When invoked, return the relevant rule(s) for the change being made and flag any direct violations you can spot in the diff. The actual overflow inspection lives in the sibling `slide-overflow-check` subagent — don't re-implement it here. + +## Slide Deck Rules + +The pptx exporter is the most visually-sensitive surface in the project. Several non-obvious rules keep its output safe for a thesis-defence audience. + +### 1. Canvas geometry (16:9 widescreen) + +- `slide_width = 13.333"`, `slide_height = 7.5"`. +- Body area sits between `BODY_TOP = 1.5"` and `FOOTER_GUARD = 7.0"`. +- Never let a shape's *rendered* text extend past `FOOTER_GUARD = 7.05"` (the line where page numbers and footers live). + +### 2. Three rendering tiers + +`PptxExporter._add_paper_slides` dispatches by inspecting `Paper.summary`: + +| Tier | Trigger | Path | +|---|---|---| +| Thesis-style | `summary.has_rich_fields()` | `_add_rich_summary_slides` — pain-point quadrant, RQ callout, KPI block, technique table, literature positioning, system overview, method details, per-RQ result tables, contribution summary, core observation, limitations & future work, Q&A, references. | +| Enriched-flat | `summary` populated only in flat tier | `_add_flat_summary_slides` — one slide per flat section (motivation / contributions / method / results / …). | +| Lightweight | no `summary` | `_add_abstract_split_slides` — cover + agenda + Background / Approach / Findings sentence buckets + references. | + +### 3. Defensive truncation + +- Every textbox runs its text through `_truncate(..., _BULLET_MAX_CHARS)`. +- Multi-column / quadrant cells use the narrower `_BULLET_MAX_CHARS_COL = 28` (half-width columns wrap sooner). +- Section titles cap at `_SLIDE_TITLE_TRUNCATE = 60` chars so 30pt fits in the two-line title box. + +### 4. Per-slide content caps + +- `_MAX_STACKS_PER_SLIDE = 5` +- `_METHOD_SECTIONS_PER_SLIDE = 2` +- `_EVALUATION_SECTIONS_PER_SLIDE = 2` +- KPI blocks and core-observation callouts are **always** split onto their own slide (`_add_kpi_slide`, separate core-observation slide). Never balance "stacks + tail callout" inside a fixed height. + +### 5. Semantic shape names + +Every textbox is named with one of: `title` / `meta` / `body` / `subhead` / `footer` / `page_number` / `kpi` / `kpi_label` / `rq_box` / `paper_subtitle`. `pptx_edit.update_slide(..., title=...)` looks them up by name; **never break this contract** — silently renaming a shape will break the MCP edit tools. + +### 6. i18n + +All template strings (section labels, "Paper N of M", "References", footer copy, "n.d." for missing years) flow through `autopapertoppt/exporters/i18n.py`. + +``` +SUPPORTED_LANGUAGES = ( + "en", "zh-tw", "zh-cn", "ja", "es", "fr", "de", "ko", + "pt", "ru", "it", "vi", "hi", "id", +) +``` + +Every language has every key — enforced by `test_every_language_has_every_key`. Untranslated locales fall back silently to `en` via `normalise_language`. + +When adding a new template string: +1. Add the key to all 14 languages in `i18n.py`. +2. Run `py -m pytest tests/exporters/test_i18n.py` to confirm the parity test stays green. + +### 7. No overflow regressions + +When changing the deck or i18n, delegate to the `slide-overflow-check` subagent — it walks every shape on every slide and checks rendered-text height vs. the box's reserved height, and confirms no shape extends past the footer guard. + +--- + +## LLM-as-agent vs Python pipeline (enrichment dispatch) + +Enrichment (PDF → structured `PaperSummary`) has two execution paths. Code MUST keep them cleanly separated. + +### Path A — LLM-as-agent (no `ANTHROPIC_API_KEY`) + +An MCP-aware LLM (e.g. Claude in this Code session) drives the workflow: +1. `fetch_paper` to get metadata. +2. `fetch_pdf_text` to extract body text. +3. LLM reads the text in-context and writes a `PaperSummary` dict. +4. `export` consumes `papers[*].summary` with the full rich-tier schema. + +No API key needed. The MCP server's `export` tool accepts the rich schema. + +### Path B — Python pipeline (`ANTHROPIC_API_KEY` set) + +The Python process calls Anthropic itself via `autopapertoppt/intelligence/summarise.py`. Auto-enrichment is the default when the env var is present. + +- `--lightweight` skips it (no API calls). +- `--enrich` flag fails loud if the env var is missing, rather than falling back. +- Default model `claude-opus-4-7`; override via `--llm-model` or `AUTOPAPERTOPPT_LLM_MODEL`. +- Requires the `[intelligence]` extra (`pypdf` + `anthropic`). + +### Rule + +Do not collapse these into a single path. The dispatch lives in `autopapertoppt/cli.py` and `autopapertoppt/intelligence/__init__.py` — keep them separate. + +**When you (the LLM) drive the session and there's no key,** rich thesis-style PPT is the default deliverable — lightweight is a fallback. **Delegate to the `paper-summary-author` subagent**, which owns the full authoring procedure (PDF reading, URL-from-xlsx rule, contributions cap, paywalled-PDF WebRunner MCP path, anti-patterns) and chains `post-author-audit` + `slide-overflow-check` before the deck ships. Do NOT tell the user "set `ANTHROPIC_API_KEY` for a rich deck" — you ARE the LLM that could write the summaries. + +--- + +## When invoked + +1. Identify which file the parent agent is editing. +2. Surface only the rules in this doc that apply (don't dump the whole doc). +3. If the diff visibly violates a rule (e.g. a textbox without `name=`, a section header > 60 chars hardcoded, a new i18n key only added to `en`), flag it: `path:line — rule — one-line summary`. +4. For overflow, defer to `slide-overflow-check`. diff --git a/.claude/agents/slide-overflow-check.md b/.claude/agents/slide-overflow-check.md new file mode 100644 index 0000000..2b0378c --- /dev/null +++ b/.claude/agents/slide-overflow-check.md @@ -0,0 +1,90 @@ +--- +name: slide-overflow-check +description: Inspect a generated .pptx for overflow regressions — every shape's wrapped-text rendered height must fit inside its box, and no shape may extend past the 7.05" footer guard on a 16:9 widescreen slide. Use after any change that touches autopapertoppt/exporters/ or autopapertoppt/exporters/i18n.py. +tools: Bash, Read, Grep, Glob +--- + +You are the slide-deck overflow inspector for the AutoPaperToPPT project. Your job is to verify that a generated `.pptx` is safe to ship to a thesis-defence audience — no shape that wraps text past its allotted box, no shape that pokes into the page-number / footer band. + +## What overflow means here + +The pptx exporter writes 16:9 widescreen slides: + +- `slide_width = 13.333"` (12192000 EMU) +- `slide_height = 7.5"` (6858000 EMU) +- Body area sits between `BODY_TOP = 1.5"` and `FOOTER_GUARD = 7.0"` +- The 0.05" buffer below the footer guard (i.e. `7.05"`) is the hard ceiling — anything beyond it visibly collides with page numbers and the footer copy. + +A shape "overflows" when its **wrapped, rendered** text height exceeds either: +1. The shape's own height (`shape.height`), causing text to spill outside its frame; OR +2. `7.05"` (6,400,800 EMU), regardless of the shape's height. + +Both must be checked. Truncation at the source (`_truncate(..., _BULLET_MAX_CHARS)`) reduces the risk but does not eliminate it — multi-column layouts wrap at narrower widths, and i18n languages (CJK, hi, vi) wrap differently than en. + +## How to run the inspection + +You'll be told (or you can infer from context) which deck(s) to check. Typical inputs: + +- A specific path: `exports//.pptx` +- Or a regen script the parent just ran: re-derive the path from the script's `out_dir` + `filename_stem`. + +For each deck, run a headless inspection that walks every slide, every shape, estimates the wrapped text height, and flags violations. The reference inspection pattern is `scripts/regen_ieee_thesis_style.py` and the report shape is in `exports/v3-final-overflow-check.txt`. If neither exists in the current repo, write the inspection inline with `python-pptx`: + +```python +from pptx import Presentation +from pptx.util import Emu + +FOOTER_GUARD_EMU = int(7.05 * 914400) # 7.05" in EMU + +def estimate_wrapped_height(shape) -> int: + """Rough wrap estimator: count lines including soft-wraps at ~chars/width.""" + # Implementation: walk paragraphs, measure font size, estimate chars-per-line + # from shape width and font, sum line heights. Project's inspector script + # already does this — prefer importing it over reinventing. + ... + +prs = Presentation(pptx_path) +violations = [] +for idx, slide in enumerate(prs.slides, start=1): + for shape in slide.shapes: + if not shape.has_text_frame: + continue + top = shape.top or 0 + height = shape.height or 0 + rendered = estimate_wrapped_height(shape) + bottom = top + rendered + if rendered > height: + violations.append((idx, shape.name, "overflows its box", rendered, height)) + if bottom > FOOTER_GUARD_EMU: + violations.append((idx, shape.name, "crosses footer guard", bottom, FOOTER_GUARD_EMU)) +``` + +Prefer reusing the project's existing inspector (look for `scripts/regen_ieee_thesis_style.py` or any `overflow_check.py`) over writing your own — it already knows the per-font-size estimation constants the project uses. + +## Reporting format + +Reply with a single fenced block per deck inspected: + +``` +overflow check — +slides: +shapes: +violations: + slide , shape "": — rendered " vs " + ... +verdict: PASS / FAIL +``` + +If `FAIL`, list every violation. Do not truncate — the parent agent needs the full list to fix the deck. + +## When to call yourself done + +- ALL inspected decks have `verdict: PASS`, OR +- You've reported every violation with enough detail (slide #, shape name, kind, measurements) for the parent to act on. + +## Things you do NOT do + +- Do not modify the deck or the exporter source. Inspection only. +- Do not "approximately" pass a deck that has a single violation. One violation is a fail. +- Do not invent the FOOTER_GUARD value — it's `7.05"` (i.e. body guard 7.0" + 0.05" buffer). If you find the codebase uses a different number, surface the discrepancy rather than silently adopting it. +- Do not check non-pptx artefacts. xlsx / bib / md have their own validators. diff --git a/.gitignore b/.gitignore index 2cdd61c..a8d780d 100644 --- a/.gitignore +++ b/.gitignore @@ -185,6 +185,23 @@ cython_debug/ # re-generated; committing them would bloat history. exports/ +# WebRunner artifacts — Chrome user-data dirs (when AUTOPAPERTOPPT_CHROME_PROFILE_DIR +# is set to a repo-local path) and partial PDF downloads. Selenium / je_web_runner +# state and screenshots are user-machine-specific and must never be committed. +chrome_profile/ +chrome-profile/ +chrome_profiles/ +selenium-debug.log +*.crdownload + # Local agent / IDE settings — user-specific, not part of the project. -.claude/ +# Exception: .claude/agents/ ships project-scoped subagent definitions (the +# four task agents — dod-verify, paper-summary-author, post-author-audit, +# slide-overflow-check — plus the four reference agents split out from +# CLAUDE.md: code-quality-reviewer, compliance-auditor, slide-deck-rules, +# env-vars). Use ".claude/*" (not ".claude/") so the directory itself is +# not ignored — otherwise git would refuse to re-include any child path +# via "!". +.claude/* +!.claude/agents/ .idea/ \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 47fffd1..1b189f5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -205,13 +205,19 @@ Default mix (no env vars required): `arxiv`, `semantic_scholar`, `openalex`, `openaire`. Pulled in automatically when `--source` is not given. Opt-in plugins (need an env var or explicit flag): -- `ieee` — set `AUTOPAPERTOPPT_IEEE_API_KEY` (official API) or - `AUTOPAPERTOPPT_ENABLE_IEEE_SCRAPING=1` (ToS-grey fallback). +- `ieee` — **on by default**, search and document fetch go through + **visible Chrome via WebRunner** (`sources/ieee/webrunner_backend.py`). + See "IEEE / paywalled domains use WebRunner" below — this is a hard + rule, not a perf hint. Set `AUTOPAPERTOPPT_IEEE_API_KEY` to switch + to the official Xplore API path; set + `AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING=1` to opt out entirely + (CI / no-Chrome environments only). - `springer` — set `AUTOPAPERTOPPT_SPRINGER_API_KEY` (free key from https://dev.springernature.com/). Required — the plugin raises `ConfigError` without it. - `scholar` — set `AUTOPAPERTOPPT_ENABLE_SCHOLAR_SCRAPING=1`. Google - Scholar ToS forbids scraping; off by default. + Scholar ToS forbids scraping; off by default. When on, also goes + through WebRunner (visible Chrome), not httpx. For top-tier-only searches (the default), the filter in `autopapertoppt/core/top_venues.py` accepts arXiv passthrough plus a @@ -232,6 +238,18 @@ flagships (Nature, Science, PNAS, CACM, Lecture Notes in CS, …). Pass generation, a paywall ratio above 30 percent prompts the user before any slides are produced. `--yes` skips the prompt. Single-paper `--paper` mode aborts with exit 1 if the PDF can't be retrieved. +- **IEEE / paywalled domains use WebRunner, not httpx.** IEEE search, + IEEE document fetch, Google Scholar search, and any paywalled-PDF + download from publisher CDNs (ieeexplore.ieee.org, dl.acm.org, + link.springer.com, sciencedirect.com, wiley/oup/nature/science/…) + MUST go through visible Chrome — the IEEE plugin's WebRunner backend, + the Scholar plugin's WebRunner backend, or `mcp__webrunner__*` tools + from the LLM-as-agent session. The httpx branch in those plugins is + a CI safety net for environments without Chrome; on a user machine + with VPN access, a silent fall-through to httpx is a bug, not an + acceptable degradation. If you don't see a Chrome window open for an + IEEE search, treat the result set as suspect. Full rule + audit + checklist: `.claude/agents/compliance-auditor.md`. - **Slide-deck guards.** 16:9 widescreen, body between 1.5" and 7.0", `FOOTER_GUARD = 7.05"`. Every textbox runs through `_truncate(...)` with the per-layout cap. Don't add slides that balance "stacks + tail @@ -247,11 +265,20 @@ flagships (Nature, Science, PNAS, CACM, Lecture Notes in CS, …). Pass ## Where to look for the rest -- Full project guide, fetcher plugin contract, exporter rules, - Definition of Done gates, lint / bandit / SonarQube rule list: - **`CLAUDE.md`**. +- Slim overview + Git Commit hygiene + Browser-Automation hard rule: + **`CLAUDE.md`** (top-level, always loaded). +- Code-quality / SOLID / linter / SonarQube rule list: + `.claude/agents/code-quality-reviewer.md`. +- Network safety, core-vs-source-plugin boundary, browser-automation + audit checklist, path-safety, suppression conventions, bandit-skip + config: `.claude/agents/compliance-auditor.md`. +- pptx rendering tiers, truncation caps, semantic shape names, i18n, + enrichment dispatch: `.claude/agents/slide-deck-rules.md`. +- Env vars + Python / `.venv` toolchain reference: + `.claude/agents/env-vars.md`. +- DoD gate runner: `.claude/agents/dod-verify.md`. +- LLM-as-agent thesis-style authoring: `.claude/agents/paper-summary-author.md` + + `post-author-audit.md` + `slide-overflow-check.md`. - Per-source plugin contract and recorded fixtures: `sources//` + `tests/fixtures//`. -- Slide-deck rendering: `autopapertoppt/exporters/pptx.py` and - `autopapertoppt/exporters/i18n.py`. - LLM-as-agent flow examples: `scripts/regen_*.py`. diff --git a/CLAUDE.md b/CLAUDE.md index d8999bd..d3e3617 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,10 +1,11 @@ # Project Guidelines -> **Other agents:** `AGENTS.md` mirrors the cross-agent must-knows -> (LLM-as-agent default path, HTTPS-only, paywall gate, slide-deck -> guards, Definition of Done). Codex CLI, recent Aider, and several -> other tools auto-load `AGENTS.md`; this file remains the canonical, -> deeper reference. Keep them in sync when you change the rules. +> **Other agents:** `AGENTS.md` mirrors the cross-agent must-knows. Codex CLI, +> recent Aider, and several other tools auto-load `AGENTS.md`; keep them in +> sync when you change rules. Detailed rules now live in `.claude/agents/` +> as subagents (`code-quality-reviewer`, `compliance-auditor`, +> `slide-deck-rules`, `env-vars`, plus the task-running agents `dod-verify`, +> `paper-summary-author`, `post-author-audit`, `slide-overflow-check`). ## Project Overview @@ -12,836 +13,108 @@ AutoPaperToPPT is a Python CLI + MCP assistant that: 1. **Searches academic papers** by user-supplied keywords across multiple sources (arXiv, Semantic Scholar, OpenAlex, PubMed, IEEE Xplore, ACM Digital Library, DBLP, - Crossref, OpenAIRE, Springer Nature, and Google Scholar via opt-in scraping). Each - source ships behind a fetcher adapter so adding a new source does not touch the - exporter layer or the MCP server. -2. **Normalises the results** into a single internal `Paper` record (title, authors, - year, venue, abstract, source URL/DOI, BibTeX key, raw payload, optional - `PaperSummary`), de-duplicates by DOI / arXiv ID / title-fuzzy-match, and ranks by - recency + citation count. -3. **Optionally enriches each paper** by fetching its PDF and producing a structured - `PaperSummary`. Two enrichment paths: - - **LLM-as-agent (no API key)** — an MCP-aware client (e.g. Claude Code) calls - `fetch_paper` + `fetch_pdf_text`, reads the body text in-context, writes a - summary dict, and passes it to `export`. - - **Python pipeline (`--enrich`)** — the CLI calls the Anthropic API itself - (`ANTHROPIC_API_KEY` required); default model `claude-opus-4-7`. -4. **Generates four outputs** from a chosen result set: - - **`.pptx` slide deck** — 16:9 widescreen, page-numbered. Three rendering paths - pick themselves based on what's present: - - *Lightweight* (abstract only) — cover + agenda + Background / Approach / - Findings sentence buckets + references. - - *Enriched-flat* (`PaperSummary` motivation/contributions/method/results/…) - — one slide per flat section. - - *Thesis-style* (`PaperSummary` rich fields) — pain-point quadrant + - research-question callout + KPI block + technique table + literature - positioning table + system overview + method details + per-RQ result - tables + contribution summary + core observation + limitations & - future work + Q&A + references. - - **`.xlsx` workbook** — "Papers" sheet with hyperlinked URL/PDF, "Query" sheet - with provenance. - - **`.bib` BibTeX file** — stable, collision-free citation keys, LaTeX-escaped. - - **`.md` summary** and **`.json` raw payload** also available as exporters. -5. **Exposes an MCP server** that surfaces every step as a tool (`search`, - `fetch_paper`, `fetch_pdf_text`, `export`, `pptx_inspect`, `pptx_update_slide`, - `pptx_delete_slide`, `pptx_reorder_slides`, `pptx_add_slide`). - -The whole stack is single-process and runs on Python 3.12+. Heavy I/O (network -fetches, PDF text extraction, LLM calls) MUST happen off the event loop's main -thread; the shared `httpx.AsyncClient` registry pools connections per source. + Crossref, OpenAIRE, Springer Nature, Google Scholar). Each source ships behind a + fetcher adapter — adding a source does not touch the exporter layer or MCP server. +2. **Normalises** results into a `Paper` record, de-duplicates by DOI / arXiv ID / + title-fuzzy-match, ranks by recency + citation count. +3. **Optionally enriches** each paper into a structured `PaperSummary` via either the + LLM-as-agent flow (no API key — MCP-aware LLM authors the summary) or the + Python pipeline (`ANTHROPIC_API_KEY` set — Anthropic API call). +4. **Generates** `.pptx` (three rendering tiers — lightweight / enriched-flat / + thesis-style), `.xlsx`, `.bib`, `.md`, `.json` outputs. +5. **Exposes** every step as an MCP tool (`search`, `fetch_paper`, `fetch_pdf_text`, + `export`, `pptx_inspect`, `pptx_update_slide`, `pptx_delete_slide`, + `pptx_reorder_slides`, `pptx_add_slide`). + +Single-process, Python 3.12+. Heavy I/O off the event loop; shared +`httpx.AsyncClient` registry pools connections per source. ### Top-level layout ``` AutoPaperToPPT/ -├── autopapertoppt/ # main package -│ ├── core/ # Paper / PaperSummary / RqResult / Query models, -│ │ # constants, dedup, ranking, pipeline, identifiers -│ ├── fetchers/ # HTTPS-only shared client, token-bucket rate -│ │ # limit, Fetcher abstract base -│ ├── exporters/ # pptx (thesis-style + lightweight), xlsx, bibtex, -│ │ # markdown, json + pptx_edit + i18n -│ ├── intelligence/ # PDF fetch/extract + Anthropic summariser -│ │ # (optional [intelligence] extra) -│ ├── mcp/ # FastMCP server registering all tools -│ ├── utils/ # logging, path safety, async helpers -│ ├── cli.py # argparse CLI -│ └── __main__.py # `python -m autopapertoppt` -├── sources// # per-source plugins (arxiv/, semantic_scholar/, -│ # openalex/, pubmed/, ieee/, acm/, scholar/, -│ # dblp/, crossref/, openaire/, springer/) -│ ├── __init__.py # sets `fetcher_class` -│ ├── fetcher.py # Fetcher subclass -│ └── parser.py # source-specific payload → Paper -├── tests/ # pytest suite + fixtures (hermetic, no live HTTP) -│ ├── fixtures//*.json|xml|html -│ └── sources/test_.py -├── docs/ # Sphinx tree (en + zh-tw + zh-cn) -├── scripts/ # one-off regen / fixture-record scripts -├── pyproject.toml # ruff, bandit, build, optional extras -└── .bandit # canonical bandit skip list +├── autopapertoppt/ # main package +│ ├── core/ # Paper / PaperSummary / Query models, dedup, ranking, pipeline +│ ├── fetchers/ # HTTPS-only shared client, token-bucket rate limit, WebRunner browser +│ ├── exporters/ # pptx (rich + lightweight), xlsx, bibtex, markdown, json + pptx_edit + i18n +│ ├── intelligence/ # PDF fetch/extract + Anthropic summariser ([intelligence] extra) +│ ├── mcp/ # FastMCP server registering all tools +│ ├── utils/ # logging, path safety, async helpers +│ ├── cli.py # argparse CLI +│ └── __main__.py +├── sources// # per-source plugins (arxiv/, semantic_scholar/, openalex/, pubmed/, +│ # ieee/, acm/, scholar/, dblp/, crossref/, openaire/, springer/) +├── tests/ # pytest + recorded fixtures (hermetic, no live HTTP) +├── docs/ # Sphinx tree (en + zh-tw + zh-cn) +├── scripts/ # one-off regen / fixture-record scripts +├── pyproject.toml # ruff, bandit, build, optional extras +└── .bandit # canonical bandit skip list ``` ## Definition of Done (HARD REQUIREMENT) -Every feature, bug fix, refactor, or behaviour change MUST satisfy ALL of the following before -it can be committed. No exceptions — incomplete work stays on the working copy until the gates -pass. - -1. **Unit tests are written and they pass.** New code without new tests is incomplete; the - commit fails this gate. See the **Unit Tests** section below for the exact coverage - expectations. -2. `py -m pytest tests/` runs clean (or only skips that already existed before the change). -3. `py -m ruff check .` reports no new errors. -4. `py -m bandit -c pyproject.toml -r autopapertoppt/ sources/` reports `No issues identified`. -5. **End-to-end smoke check** for any change that touches `sources/`, - `autopapertoppt/exporters/`, `autopapertoppt/intelligence/`, or - `autopapertoppt/mcp/`: - - Run `py -m autopapertoppt --query "transformer attention" --source arxiv - --max 3 --out ./exports/smoke/` and confirm `.pptx`, `.xlsx`, `.bib` land - on disk and the deck opens without warnings (no overflow into the footer). - - For pptx changes, also run an enriched / thesis-style regen against a - known paper (see `scripts/regen_*.py`) and inspect the output with a - headless slide-overflow check — every shape's rendered text height must - fit within its allotted box, no shape may extend past 7.05" on a 16:9 - widescreen slide. - - For MCP changes, hit `python -c "from autopapertoppt.mcp import - build_server; import asyncio; print(asyncio.run(build_server().list_tools()))"` - and verify every documented tool is present. -6. **No live network calls in tests** — every fetcher test uses recorded fixtures - (`tests/fixtures//*.json|html`). Recording new fixtures is a separate, manual step - (`scripts/record_fixture.py`) and the recorded file is committed. -7. The commit message contains no AI tool/model names and no `Co-Authored-By` line. - -When you finish editing code, work through this list explicitly before staging. If a gate -fails, fix it — do not ship around it. Skipping tests "to come back later" is not allowed -because later never happens and the gap compounds. +Every change MUST pass the full gate set before commit. **Delegate to the +`dod-verify` subagent** — it owns the exact gate list, commands, and pass/fail +report format (and chains `slide-overflow-check` when exporters/i18n change, +`code-quality-reviewer` for deeper code-quality review, +`compliance-auditor` for project conventions). Skipping a gate "to come back +later" is not allowed. ## Git Commits -- NEVER add `Co-Authored-By` lines to commit messages. All commits should only contain the - commit message itself with no co-author attribution. -- NEVER mention "Claude", "Claude Code", "AI-generated", "GPT", "Copilot", or any AI tool / - model name anywhere — including commit messages, PR titles, PR descriptions, code - comments, and documentation. - -## Code Quality Requirements - -### Design Patterns - -- Apply appropriate design patterns (Strategy, Adapter, Factory, Observer, Command, Builder, - Decorator, Template Method) where they fit naturally. Fetchers are Strategies behind a - Factory; exporters are Strategies; the search pipeline is a Chain of Responsibility - (fetch → normalise → dedup → rank → cache); rate limiting is a Decorator on the HTTP - client. -- Prefer composition over inheritance. A `Paper` is a dataclass of fields + a `RawPayload` - attachment, not a deep class hierarchy. -- Follow SOLID principles: Single Responsibility, Open/Closed, Liskov Substitution, Interface - Segregation, Dependency Inversion. The exporter layer depends on the `Paper` / - `PaperCollection` interfaces, never on a concrete fetcher's response shape. -- Apply DRY — extract shared HTTP / rate-limit / retry logic into `autopapertoppt/fetchers/`; - never copy a `requests`/`httpx` setup across source plugins. -- Reuse the existing project patterns: `httpx.AsyncClient` for network, `asyncio.Semaphore` - for per-source concurrency caps, FastAPI dependency injection for the cache + settings, - Streamlit `st.session_state` (never module globals) for UI state. - -### Software Engineering Practices - -- Separate concerns: the exporters never call the network — they consume an in-memory - `PaperCollection`. The UI never parses HTML — it receives normalised `Paper` records from - the API layer. -- Write self-documenting code with clear naming; add comments only for non-obvious "why" - explanations (e.g. "Google Scholar returns no stable ID, so we hash title+first-author+year - to form the dedup key"). -- Favor immutability where practical — `Paper`, `Query`, and `ExportRequest` are frozen - dataclasses; mutations create a new instance. -- Handle errors explicitly at system boundaries (network calls, file IO, HTML parsing, - exporter rendering); propagate exceptions cleanly through internal layers. Wrap every - HTTP call in a helper that raises a typed `FetchError` (`RateLimitError`, `ParseError`, - `SourceUnavailableError`) — never swallow. -- Keep functions short and focused — one function, one responsibility. -- Delete dead code immediately; do not comment it out or leave unused imports/variables. - -### Performance - -- Always consider and implement the best-performance approach for the task. -- Use lazy loading and on-demand initialization where applicable. Fetcher plugins are - imported on first use, not at app startup; the pptx template is parsed once and cached. -- Avoid unnecessary memory allocations and copies — stream large response bodies through - `httpx` rather than loading entire HTML pages into memory when only a result list is - needed. -- Prefer batch operations over per-item processing. Group fetches by source, run sources in - parallel with `asyncio.gather`, but cap per-source concurrency with a semaphore. -- Use appropriate data structures: dict for O(1) DOI / arXiv-ID lookup, set for the dedup - key set, deque for the rate-limit token bucket history, dataclasses for hot record paths. -- Profile and measure before optimizing hot paths; avoid premature optimization of cold - paths. `autopapertoppt/utils/profiling.py` exposes `with section("name"):` — use it before - claiming a perf win. -- Cache expensive operations with `functools.lru_cache` (in-process) or the disk cache in - `autopapertoppt/cache/` (cross-run). Every raw network response is cached keyed by - `sha256(source + normalized_query + page)`; default TTL is configurable per source. -- Use generators / `AsyncIterator` for large result pages so the UI can start rendering the - first page before later pages arrive. -- Never block the event loop with synchronous network calls. Use `httpx.AsyncClient`, not - `requests`. Synchronous `requests` is allowed ONLY in the fixture-recording script. - -### Async & Concurrency Rules - -- The FastAPI process owns **exactly one** `httpx.AsyncClient` per source, created at - startup and reused for the whole process lifetime. Do NOT create a fresh client per - request — connection pooling and rate-limit token-bucket state must persist. -- Per-source rate limits live in `autopapertoppt/fetchers/rate_limit.py` as a token-bucket - decorator. Each source plugin declares its own bucket (`arxiv: 1 req/3s`, - `semantic_scholar: 1 req/s`, `scholar: 1 req/10s with jitter`, etc.). Do NOT bypass the - bucket — even retries go through it. -- Streamlit runs the UI on a separate thread per session. Mutate `st.session_state` only, - never module globals. Long-running export jobs are dispatched to the FastAPI backend and - polled, not run inline in the Streamlit script. -- All fixture-recording, CLI exports, and tests use `asyncio.run` at the outermost layer - and never inside library code. - -### Security - -- Never hardcode secrets, API keys, tokens, or passwords in source code — use environment - variables (`AUTOPAPERTOPPT_IEEE_API_KEY`, `AUTOPAPERTOPPT_SCHOLAR_PROXY`, …) loaded via - `pydantic-settings`. Document required env vars in `README.md`. -- Validate and sanitize ALL external input (user keywords, API responses, scraped HTML, - uploaded BibTeX) at system boundaries. Strip control characters from keywords before - building URLs; cap query length to a configurable hard limit. -- Sanitize file paths to prevent path traversal — every export `out` path is resolved - through `autopapertoppt/utils/path_safety.py::resolve_safe(root, reference)`, which rejects - `..` segments, rejects absolute paths from the API request body, and asserts the resolved - path stays under `root`. -- Apply the principle of least privilege — fetcher plugins only see the curated HTTP client - and a logger. They never see the filesystem, the cache layer, or other sources' - credentials. -- Avoid `eval()`, `exec()`, `pickle.loads()` on untrusted data, and `subprocess` with - `shell=True`. Cached payloads are stored as JSON or msgpack, never pickle. -- All network traffic uses HTTPS. The shared HTTP client rejects any URL whose scheme is - not `https` via the `_https_only_transport` wrapper (see **Network Safety** below). -- Respect robots / ToS — Google Scholar scraping is OFF by default and must be opted in by - setting `AUTOPAPERTOPPT_ENABLE_SCHOLAR_SCRAPING=1`. Per-source `User-Agent`, request - spacing, and concurrency caps are declared in each plugin and MUST NOT be removed to "make - things faster". -- Use secure defaults: SHA-256 for cache keys and dedup hashes, `secrets.token_urlsafe` for - any session token, constant-time comparisons for any future signature checks. -- Log security-relevant events (rejected URLs, malformed responses, rate-limit hits) but - never log full API keys or full HTML response bodies — truncate to 256 chars and redact - any token-shaped strings. - -### Unit Tests - -Tests are not optional polish — they are part of the change. A feature without tests is an -incomplete feature and MUST NOT be committed. This rule applies equally to bug fixes -(regression test required) and refactors (existing behaviour must remain green; add a test -if the refactor exposes a previously untested path). - -**Required coverage for every change:** - -- **Happy path** — the new code does what it advertises on a representative input (a small - recorded arXiv response, a 2-result PubMed XML, a single-page Scholar HTML snapshot). -- **Edge cases** — empty result sets, single-paper sets, missing optional fields (no DOI, - no abstract, no year), Unicode-heavy titles, multi-author truncation, duplicate papers - across sources. -- **Error handling** — every `except` branch is exercised; HTTP 429 raises - `RateLimitError`; malformed JSON / HTML raises `ParseError`; an exporter writing to an - unwritable path raises `ExportError`. -- **Boundary conditions** — values just inside and just outside any limit (max keyword - length, max results per page, min/max year filter, BibTeX key collision counter). -- **Round-trips** — `Paper.to_dict → from_dict → equal`; `BibTeX render → parse → equal`; - cache write → cache read → equal. - -**Required test types for every feature:** - -- **Pure-helper tests.** Extract pure logic out of network / IO classes (dedup hashing, - ranking, BibTeX key generation, abstract cleaning) into helper modules and unit-test - those directly without spinning up `httpx` or FastAPI. Cheap, fast, deterministic. -- **Fetcher tests against recorded fixtures.** Every source plugin has a - `tests/sources//test_.py` that loads - `tests/fixtures//.json|html` via a monkeypatched transport and asserts - the parsed `Paper` list. No live network calls — the test suite must be runnable - offline. -- **API tests.** Use FastAPI's `TestClient` to call `/search`, `/export`, `/status` with - the fetcher layer monkeypatched to a fake that returns canned `Paper` records. -- **UI smoke test.** Use `streamlit.testing.v1.AppTest` to drive the Streamlit page, - enter a keyword, click search, and assert the results table renders. Long-running - features get their own AppTest scenario. -- **Exporter tests.** Render to a `tmp_path`, then re-open the artefact and assert - structure: `python-pptx` to assert slide count + title text; `markdown-it` (or plain - string match) for `.md`; `bibtexparser` for `.bib`. Binary PDF tests assert non-empty - file + valid `%PDF-` magic header. -- **Integration test where the wiring is non-obvious.** End-to-end fetch → dedup → rank → - export on a small recorded multi-source fixture. - -**Mechanics:** - -- Use `pytest` + `pytest-asyncio` style. Module-level functions and `Test*` classes are - both fine; follow the style of the file you're adding to. -- Test file naming: `tests/test_.py` for core, `tests/sources//...` for - fetchers, `tests/exporters/test_.py` for exporters. -- Use the shared fixtures in `tests/conftest.py` (`http_recorder`, `fake_cache`, - `sample_papers`, `tmp_export_root`). Do not roll your own async loop or `httpx` client. -- Tests that need a recorded HTTP exchange use the `http_recorder` fixture, which loads - the matching JSON/HTML file and asserts the request URL + headers match what was - recorded. Re-recording is a manual step (`scripts/record_fixture.py`) — never let a - test silently mutate fixtures. -- Never write to the user's real cache or settings file. The autouse - `_isolate_user_paths` fixture redirects `AUTOPAPERTOPPT_CACHE_DIR` and - `AUTOPAPERTOPPT_CONFIG_DIR` to `tmp_path`. -- Run `py -m pytest tests/` before committing. If a test was already skipping because of - a missing optional dependency, leave it skipping — but every NEW test must run, not - skip. - -### Linter & Static Analysis Compliance (SonarQube / Codacy / pylint / flake8 / ruff) - -All new and modified code MUST pass the following rules without warnings. These mirror the -default rule sets of SonarQube, Codacy, pylint, flake8, ruff, and bandit for Python. - -#### Complexity & Size - -- **Cognitive complexity**: keep each function ≤ 15 (SonarQube `python:S3776`). Break - nested branches into helper functions when exceeded. -- **Cyclomatic complexity**: keep each function ≤ 10 (pylint `R1260`, radon `C`). -- **Function length**: ≤ 75 logical lines. Split long functions into focused helpers. -- **File length**: ≤ 1000 lines (SonarQube `python:S104`). Split large modules. -- **Parameter count**: ≤ 7 per function (SonarQube `python:S107`). Group related params - into a dataclass when exceeded. (`Query`, `ExportOptions`, `FetcherConfig`.) -- **Nesting depth**: ≤ 4 levels (SonarQube `python:S134`). Use early returns / guard - clauses. -- **Boolean expression complexity**: ≤ 3 operators in one expression (SonarQube - `python:S1067`). Extract to named booleans. -- **Return statements**: ≤ 6 per function (pylint `R0911`). -- **Local variables**: ≤ 15 per function (pylint `R0914`). - -#### Duplication - -- Do NOT copy-paste blocks of ≥ 3 statements across functions or files (SonarQube - `common-python:DuplicatedBlocks`, Codacy duplication detector). Extract shared logic — - HTTP setup, retry/backoff, abstract cleaning, BibTeX key generation all live in one - place. -- Do NOT declare the same string literal ≥ 3 times (SonarQube `python:S1192`). Assign to - a module-level constant. Source names (`"arxiv"`, `"pubmed"`, …) live in - `autopapertoppt/core/sources.py` constants. - -#### Naming (PEP 8) - -- `snake_case` for functions, methods, variables, modules (SonarQube `python:S1542`, - pylint `C0103`). -- `PascalCase` for classes (pylint `C0103`). -- `UPPER_CASE_WITH_UNDERSCORES` for module-level constants. -- `_leading_underscore` for private attributes / methods. -- No single-letter names except loop indices (`i`, `j`, `k`) or well-known short forms - (`q` for query in obvious local scope, `r` for response in a `with httpx.stream(...)` - block). - -#### Errors & Exceptions - -- Never use bare `except:` — always specify the exception type (SonarQube `python:S5754`, - flake8 `E722`). -- Never write `except Exception: pass` without a logged reason and comment explaining why - it is safe. -- Never catch `BaseException` directly (covers `KeyboardInterrupt`, `SystemExit`). -- Raise specific exception types — define a domain hierarchy: `AutoPaperToPPTError` → - `FetchError` (`RateLimitError`, `ParseError`, `SourceUnavailableError`), `CacheError`, - `ExportError`, `ConfigError`. -- Chain exceptions with `raise X from err` to preserve context (ruff `B904`). -- Never use `assert` for runtime validation (assertions are stripped under `python -O`); - use explicit `raise` instead. `assert` is only for invariants in tests. - -#### Code Smells - -- No unused imports, variables, or function parameters (pyflakes `F401`, `F841`, pylint - `W0612`, `W0613`). Prefix intentionally unused params with `_`. -- No commented-out code. Delete it — git preserves history. -- No `print()` calls in production code; use the project's logger - (`autopapertoppt/utils/logging`). -- No `TODO` / `FIXME` / `XXX` left in merged code (SonarQube `python:S1135`). File a - ticket instead. -- No magic numbers — extract to `UPPER_CASE` constants (SonarQube `python:S109`). - Exceptions: `0`, `1`, `-1`, `2` in obvious contexts. Common constants - (`DEFAULT_PAGE_SIZE = 25`, `MAX_RESULTS_PER_SOURCE = 200`, `CACHE_TTL_SECONDS = 86400`) - live in `autopapertoppt/core/constants.py`. -- Use `is None` / `is not None` (never `== None` / `!= None`) (pycodestyle `E711`). -- Use `isinstance(x, T)` instead of `type(x) == T` (pycodestyle `E721`). -- No mutable default arguments (`def f(x=[])`) — use `None` and assign inside (ruff - `B006`, pylint `W0102`). -- No global mutable state; if unavoidable, encapsulate in a module-level class or - singleton (the shared HTTP client registry, the cache handle, the rate-limit buckets). -- Prefer f-strings over `.format()` or `%` (ruff `UP032`). -- Always use context managers (`with` / `async with`) for file / HTTP / DB resource - handles (ruff `SIM115`). -- Prefer `dict.get(key, default)` over `if key in dict: ... else: ...` (ruff `SIM401`). -- Use comprehensions / generator expressions instead of `map` + `lambda` or manual - `append` loops when clearer. - -#### Security (bandit / SonarQube `python:S*` security rules) - -- `pickle.load(s)` on untrusted data is forbidden (`B301`, SonarQube `python:S5135`). - Cache payloads are JSON or msgpack with a strict schema. -- `yaml.load` without `SafeLoader` is forbidden — use `yaml.safe_load` (`B506`). -- MD5 / SHA-1 are forbidden for security purposes — use SHA-256+ (`B303`, `B304`, - SonarQube `python:S4790`). Allowed for non-security uses (cache keys, dedup hashes) - ONLY with `usedforsecurity=False`. -- `subprocess` with `shell=True` is forbidden when any argument comes from user input - (`B602`). The PDF export shells out to `pandoc` / `weasyprint` via the args-list form - only. -- Never use `eval`, `exec`, `compile` on dynamic input (`B307`). There are no exceptions - in this project. -- Never use `tempfile.mktemp()` — use `tempfile.mkstemp()` or `NamedTemporaryFile` - (`B306`). -- Network binds must not use `0.0.0.0` unless intentional and documented (`B104`). The - FastAPI app defaults to `127.0.0.1`. -- XML parsing (PubMed XML, arXiv Atom feed) MUST use `defusedxml`, never stdlib - `xml.etree` on untrusted input (`B405`–`B411`). -- HTML parsing uses `beautifulsoup4` with `lxml` parser; never `eval`-style attribute - evaluators. -- Random number generation for security must use `secrets`, not `random` (`B311`). - Backoff jitter MAY use `random` and should pin a seed in tests for reproducibility. -- All `urlopen` / `httpx` calls go through the project HTTPS-only transport (see - **Network Safety** below). Direct `requests.get` / `urllib.request.urlopen` is - forbidden in production code. - -#### Typing & Documentation - -- Public functions and methods MUST have type hints on parameters and return type. Use - `pydantic` models or `dataclasses` for structured payloads; `list[Paper]`, not bare - `list`. -- Public modules and classes SHOULD have a one-line docstring describing their purpose. -- Private helpers may omit docstrings if names are self-explanatory. -- Each source plugin's `fetcher.py` carries a module docstring stating the source name, - the endpoint(s) it talks to, the rate limit, and whether an API key is required. - -#### Enforcement - -When writing or modifying code, mentally check each function against the above rules -before finalising. If unavoidable rule violation (e.g. a FastAPI dependency signature -forces extra parameters, or a parser genuinely needs a long match block), add a -`# noqa: ` or equivalent suppression with a brief justification comment on the -same line. - -## Project-Specific Compliance Patterns - -### Core vs Source Plugins - -The line between `autopapertoppt/` (the main package) and `sources//` is **not** -"anything source-related goes in sources" — it's **dependency surface and failure -isolation**. - -**A feature is a source plugin when ANY of the following is true:** - -1. It needs a **heavy / optional runtime dependency** that we don't want to force on every - user (e.g. `selenium` for Scholar JS-rendered pages, `xmltodict` for PubMed, a vendor - SDK for IEEE Xplore). -2. It needs **failure isolation** — a flaky third-party API or scraping target should - never bring down the search pipeline. Other sources keep returning results. -3. It needs **independent release cadence** — a Scholar HTML layout change can be patched - without re-shipping the core engine. - -**A feature stays in the core when:** - -- It runs on the default dep set (`httpx`, `pydantic`, `defusedxml`, `python-pptx`, - `openpyxl`, `bibtexparser`, `beautifulsoup4`, `lxml`, `markdown-it-py`). -- It's part of the everyday search / export workflow that all users should see by - default (arXiv, Semantic Scholar, PubMed are core; Scholar scraping and IEEE - scraping are opt-in plugins gated by env vars; ACM via Crossref is a plugin). - -#### Optional extras (opt-in installs) - -| Extra | Pulled in | Why optional | -|---|---|---| -| `[intelligence]` | `pypdf`, `anthropic` | PDF text extraction + Anthropic API for `--enrich`. Not needed for the LLM-as-agent flow over MCP. | -| `[mcp]` | `mcp` SDK | Only for users who want to run / register the MCP server. | -| `[web]` | `fastapi`, `uvicorn`, `streamlit` | Reserved for the future web UI. CLI + MCP do not need it. | -| `[dev]` | All of the above + `pytest*`, `ruff`, `bandit` | Developer toolchain. | - -#### Directory rules - -- **Core**: `autopapertoppt//.py` for pure logic. -- **Source plugin**: `sources//__init__.py` (sets `fetcher_class`), - `sources//fetcher.py` for the adapter, and **all source-internal parsing / - HTML-specific logic lives INSIDE the source directory**. Never put HTML selectors - or vendor SDK calls under `autopapertoppt/core/`. -- **Intelligence**: `autopapertoppt/intelligence/pdf.py` and `summarise.py` are - lazy-imported behind the `[intelligence]` extra. They MUST not be imported at - module top-level by any non-intelligence file. -- **Recorded fixtures**: `tests/fixtures//.{json,html,xml}`. - Re-record with `scripts/record_fixture.py --source --query "..."`. Strip - any user-specific tokens before committing. - -#### Testing source-internal modules - -Source plugins are not on the default `sys.path` — at runtime -`autopapertoppt/app/source_manager.py` prepends `sources/` so each source folder becomes -importable as a package. `tests/conftest.py` mirrors that injection at session-collect -time, which lets tests in `tests/` import source modules with -`from . import …`. Do not duplicate the path injection in individual test -files. - -#### When in doubt - -Ask: "if a user installs AutoPaperToPPT with the default `requirements.txt` and never -enables a source plugin, should this source work?" If yes → core. If no → source plugin. - -### Network Safety - -- **All outbound HTTP MUST go through `autopapertoppt/fetchers/http.py::get_client(source)`.** - It returns a per-source `httpx.AsyncClient` configured with: HTTPS-only transport, - source-specific `User-Agent`, source-specific rate-limit decorator, exponential backoff - with jitter on 429 / 5xx, and a hard total-timeout. -- Do NOT call `httpx.get` / `requests.get` / `urllib.request.urlopen` directly in new - code. Import `get_client` instead. -- The HTTPS-only transport rejects any URL whose scheme is not `https`. If a source's - documented endpoint is `http`, fix the source's config — do not bypass the transport. -- Any redirect chain that crosses to a non-`https` scheme is rejected mid-flight. -- Per-source rate limits are declared in `sources//config.py` as a `RateLimit` - dataclass (`requests_per_second`, `burst`, `jitter_seconds`). Tests assert the configured - values against the source's published policy. -- Mirror the `# nosec` pattern only where genuinely necessary: any direct `urlopen` left - for the fixture-recording script carries `# nosec B310 # scheme validated above` and - is gated by an `if scheme != "https": raise` check immediately before the call. - -### Query & Input Safety - -- User keywords are passed through `autopapertoppt/core/query.py::normalize_query` before - being embedded in any URL or body. It strips control characters, normalises Unicode - (NFC), caps length, and HTML/URL-encodes per the target source's rules. -- Date ranges, year filters, and result-count limits are validated at the FastAPI layer - with `pydantic` `Field` constraints. Out-of-range values return HTTP 422, never silent - clamping deep inside a fetcher. -- BibTeX uploads (for "import existing bibliography" features) are parsed with - `bibtexparser` in strict mode, capped at a size limit, and rejected on schema violation. - -### Export Path Safety - -- Every export `out_dir` from the CLI / MCP is resolved through - `autopapertoppt/utils/path_safety.py::ensure_export_dir(...)` and - `safe_filename(...)`. -- Filenames inside the export root are derived from a sanitised slug of the query + - timestamp (`{slug}-{YYYYMMDD-HHMMSS}.pptx`). Never use raw user-supplied filenames. - -### Slide Deck Rules - -The pptx exporter is the most visually-sensitive surface in the project. Several -non-obvious rules keep its output safe to ship to a thesis-defence audience: - -1. **16:9 widescreen.** `slide_width = 13.333"`, `slide_height = 7.5"`. Body area - sits between `BODY_TOP = 1.5"` and `FOOTER_GUARD = 7.0"`. Never let a shape's - *rendered* text extend past `FOOTER_GUARD` — that's the line at which page - numbers and footers live. -2. **Three rendering tiers.** `PptxExporter._add_paper_slides` dispatches by - inspecting `Paper.summary`: - - `summary.has_rich_fields()` → `_add_rich_summary_slides` (thesis-style). - - `summary` populated only in the flat tier → `_add_flat_summary_slides`. - - No summary → `_add_abstract_split_slides` (sentence-bucketing fallback). -3. **Defensive truncation.** Every textbox runs its text through `_truncate(..., - _BULLET_MAX_CHARS)`; multi-column / quadrant cells use the narrower - `_BULLET_MAX_CHARS_COL = 28` because half-width columns wrap sooner. Section - titles cap at `_SLIDE_TITLE_TRUNCATE = 60` chars so 30pt fits in the - two-line title box. -4. **Per-slide content caps.** `_MAX_STACKS_PER_SLIDE = 5`, - `_METHOD_SECTIONS_PER_SLIDE = 2`, `_EVALUATION_SECTIONS_PER_SLIDE = 2`. KPI - blocks and core-observation callouts are ALWAYS split onto their own slide - (`_add_kpi_slide`, separate core-observation slide). Never balance "stacks - + tail callout" inside a fixed height. -5. **Semantic shape names.** Every textbox is named `title` / `meta` / `body` / - `subhead` / `footer` / `page_number` / `kpi` / `kpi_label` / `rq_box` / - `paper_subtitle`. `pptx_edit.update_slide(..., title=...)` looks them up by - name; never break the contract. -6. **i18n.** All template strings (section labels, "Paper N of M", "References", - footer copy, "n.d." for missing years) flow through - `autopapertoppt/exporters/i18n.py`. `SUPPORTED_LANGUAGES = ("en", "zh-tw", - "zh-cn", "ja", "es", "fr", "de", "ko", "pt", "ru", "it", "vi", "hi", - "id")` — every language has every key, enforced by the - `test_every_language_has_every_key` test. Untranslated locales fall back - silently to `en` via `normalise_language`. -7. **No overflow regressions.** When changing the deck, run a headless - text-fit check that estimates the wrapped-text height of every shape (see - `scripts/regen_ieee_thesis_style.py` for an example deck and - `exports/v3-final-overflow-check.txt` for the inspection format). - -### LLM-as-agent vs Python pipeline - -Enrichment (PDF → structured `PaperSummary`) has two execution paths and code -MUST keep them cleanly separated: - -- **LLM-as-agent**: an MCP-aware LLM (e.g. Claude in this Code session) drives - the workflow. The MCP tools `fetch_paper` + `fetch_pdf_text` give it - metadata + body text; the LLM produces the structured summary in its own - context window; `export` writes the artefacts. **No `ANTHROPIC_API_KEY` is - required.** The MCP server's `export` tool understands `papers[*].summary` - with the full rich-tier schema. -- **Python pipeline (auto-on when ANTHROPIC_API_KEY is set)**: the Python - process calls Anthropic's API itself via - `autopapertoppt/intelligence/summarise.py` and produces a rich - thesis-style deck. Auto-enrichment is the **default** when the env var - is present; pass `--lightweight` to skip it and `--enrich` to fail-loud - rather than fall-back-quietly when the extras aren't installed. Default - model is `claude-opus-4-7` (override via `--llm-model` or - `AUTOPAPERTOPPT_LLM_MODEL`). Requires the `[intelligence]` extra. - -Do not collapse these into a single path. The LLM-agent flow is the cheaper -default for interactive MCP use; the Python pipeline is for unattended -automation where no LLM is otherwise in the loop. - -**Preferred path when an LLM is in the loop (CRITICAL).** Rich -thesis-style PPT is the default deliverable. Lightweight is a fallback, -never the goal when an LLM agent is in the loop. - -Decision tree: - -1. `ANTHROPIC_API_KEY` set? → CLI auto-enriches; just run it. -2. No key but you (an LLM agent) drive the session → **you write the - rich summary yourself**. The lightweight per-paper `.pptx` the CLI - just emitted is an intermediate artefact, not the deliverable. Read - each PDF, hand-author a `PaperSummary` with rich-tier fields, drop a - `scripts/regen_.py`, run it. -3. No LLM in the loop (CI / cron / unattended) → lightweight is - acceptable. - -Anti-patterns (do NOT): - -* Tell the user "set `ANTHROPIC_API_KEY` for a rich deck" while you - yourself are the LLM that could write the summaries. You are the - agent precisely so they don't have to acquire a separate key. -* Treat the per-paper lightweight `.pptx` as the final deliverable. -* Stop after `download_pdfs` reports N PDFs saved — that is the start - of the rich-authoring phase, not the end. -* Invent numbers, RQs, contributions, or limitations that don't trace - back to the paper's text. -* Fabricate `url` / `doi` / `arxiv_id` from memory when hand-authoring - a `Paper`. Publisher URL paths cannot be guessed (AAAI uses numeric - IDs like `v40i5.37389`, not author slugs; IEEE uses `arnumber`; ACM - uses opaque DOIs). Always copy these fields verbatim from the search - xlsx — see "URL / DOI verification" below. -* Leave irrelevant downloads in the run directory. The search keyword - matching is keyword-based — a query like "code review" can return a - paper on object detection literature review; "Claude code" can match - a Viterbi decoder paper because both contain "code". Once you read - the abstracts and decide a paper is off-topic for the user's actual - intent, **delete `exports//pdfs/.pdf` and the lightweight - `exports//.pptx`** so the run dir cleanly reflects the - deliverable. Keep the aggregate xlsx / bib intact — those are the - honest record of what the search returned. See "Pruning irrelevant - downloads" below for the concrete procedure. - -Worked example: `scripts/regen_llm_security_batch.py` ships 7 -hand-authored rich summaries built exactly this way. - -Per-paper flow: - -1. Get the PDF into the exports dir, one of two ways: - * `--paper ` fetches metadata and downloads the PDF - (`exports//pdfs/.pdf` lands automatically); or - * `--pdf ` when the user supplied a PDF themselves — - the file is copied into `exports//pdfs/` and a single-paper - collection is built with `source="local"`. Use `--title --authors - --year --venue --doi --arxiv-id` to override metadata when the - filename heuristic isn't right. -2. Read the PDF yourself. If the body is too large for the editor's Read - tool, run `pypdf` via the project's `intelligence.pdf._extract_text` to - dump plain text, then chunk it. Do not re-implement PDF extraction. -3. Hand-author a `PaperSummary` populated with the rich-tier fields - (`pain_points`, `research_question`, `contributions_detailed`, - `headline_metrics`, `technique_table`, `method_sections`, - `evaluation_sections`, `system_flow`, `research_questions`, - `rq_results`, `core_observation`, `limitations`, `future_work`) — only - include numbers / claims that appear verbatim in the paper. Set - `model=" (LLM-as-agent, read N-page PDF)"` and - `raw_text_chars` to the extracted length so provenance is visible on - the deck. -4. Call the exporter — either via the MCP `export` tool (when running - against a live MCP server) or directly in Python by constructing a - `Paper` with `summary=…`, wrapping it in a `PaperCollection`, and - passing it to `export_collection(...)`. Save the script under - `scripts/regen__.py` so the regen is reproducible. -5. **Canonical filename, no `-rich` suffix.** Set - `filename_stem=paper.bibtex_key()` so the rich deck overwrites the - CLI's lightweight emit at the same path. One `.pptx` per paper, the - rich one. Do not keep both — lightweight is not a deliverable. - Language variants are the only exception (e.g. `f"{key}-zh-tw"`). -6. Cap `contributions_detailed` at ≤ 4 entries (the contributions slide's - stack layout overshoots the 7.05" footer guard above that). Run the - headless overflow check from the **Slide Deck Rules** section before - handing the deck back. - -Working templates: `scripts/regen_llm_security_batch.py` (batch, 7 -papers), `scripts/regen_ling2026_agent_skills.py` (single paper en), -`scripts/regen_ling2026_agent_skills_zh_tw.py` (single paper zh-tw), -`scripts/regen_ieee_thesis_style.py` (single paper). - -#### URL / DOI verification (mandatory before handing the deck back) - -Publisher URL paths **cannot be guessed**. The author-slug pattern an -agent might invent (`view/fang2026`) is never the real AAAI URL — -AAAI uses numeric IDs (`v40i5.37389`); IEEE uses an opaque `arnumber`; -ACM uses opaque DOIs like `10.1145/3411764.3445005`. A fabricated URL -in the slide deck is worse than no URL — it visibly points the user -at a 404. - -The rule: when hand-authoring a `Paper`, copy `url` / `doi` / -`arxiv_id` **verbatim from the same search's xlsx**. Never write them -from memory; never construct them from the title. - -Concrete workflow: - -1. Run the user's search: - `py -m autopapertoppt --query "..." --out ./exports//`. - The aggregate xlsx is written to - `exports//-.xlsx` with columns - `# | Title | Authors | Year | Source | Indexed via | DOI | URL | PDF | Citations | Abstract`. -2. For every paper you author a `PaperSummary` for, copy: - * **column 7 (DOI)** → `Paper.doi` - * **column 8 (URL)** → `Paper.url` - * extract the arXiv id from column 8 when the URL is on `arxiv.org` - * leave any empty column as `None` — do NOT fabricate to fill it. -3. Strip a trailing `v1` / `v2` version suffix from arxiv URLs: - `https://arxiv.org/abs/2506.09580v1` → `arxiv_id="2506.09580"`, - `url="https://arxiv.org/abs/2506.09580"`. -4. After the regen script finishes, audit `Paper.url` vs. the xlsx - column 8 for every entry — any mismatch beyond a version suffix is - a fabrication and must be fixed before the deck ships: - - ```python - from openpyxl import load_workbook - from scripts.regen_ import ALL_PAPERS - real = {sh.cell(row=r, column=2).value: sh.cell(row=r, column=8).value - for sh in [load_workbook("exports//-.xlsx")["Papers"]] - for r in range(2, sh.max_row + 1)} - for p in ALL_PAPERS: - actual = next((u for t, u in real.items() - if p.title[:30] in (t or "")), None) - if actual and not (p.url == actual - or p.url.split("v")[0] == actual.split("v")[0]): - print(f"! {p.bibtex_key()} authored {p.url} vs real {actual}") - ``` - - This audit caught two fabrications in `regen_llm_security_batch.py` - (Wen 2025: wrong AAAI volume; Fang 2026: invented `view/fang2026` - path) before the user noticed. Re-run it whenever you add a new - paper to a regen script. - -#### Pruning irrelevant downloads (mandatory before handing the deck back) - -The search engine is keyword-based, so off-topic papers will slip in: -a query like "Claude code" returned a Viterbi decoder paper because -both contain "code"; "LLM code review" returned a paper on object -detection literature review for the same reason. Once you read the -abstracts and decide a paper is **off-topic for the user's actual -intent**, prune the run directory: - -```python -from pathlib import Path - -run_dir = Path("exports/") -irrelevant_keys = ( - "key-of-off-topic-paper-1", - "key-of-off-topic-paper-2", -) -for key in irrelevant_keys: - for path in (run_dir / "pdfs" / f"{key}.pdf", - run_dir / f"{key}.pptx"): - if path.exists(): - path.unlink() -``` - -What to delete: - -- `exports//pdfs/.pdf` — the downloaded PDF -- `exports//.pptx` — the CLI's lightweight emit - -What to **keep**: - -- The aggregate `exports//-.xlsx` and `.bib` — - they are the honest record of what the search returned. Pruning them - would rewrite history. Off-topic papers staying in the xlsx is fine - because the user can see the full search outcome there. -- The rich `*-zh-tw.pptx` / `*.pptx` files for the *relevant* papers - you hand-authored. - -Decision rule: a paper is off-topic when its actual research question -doesn't match the user's intent. "Claude (Sonnet 4.6) across six -languages" is off-topic for a "Claude Code code review" query because -the paper is about Claude the model's multilingual ability, not the -Claude Code agentic tool. Borderline cases get a rich summary (better -to over-include than to silently drop a possible match). - -### Suppression Comment Conventions - -Use the right comment for the right tool. They are NOT interchangeable. - -| Tool | Comment form | Placement | Notes | -|---------------|-----------------------------------------|-------------|-----------------------------------------------------| -| ruff / flake8 | `# noqa: ` (e.g. `# noqa: S310`) | line-level | Must list specific codes — never bare `# noqa`. | -| bandit | `# nosec B` (e.g. `# nosec B310`) | line-level | ruff's `# noqa` does NOT suppress bandit. | -| SonarCloud | `# NOSONAR` | line-level | Use for hotspots that cannot be config-skipped. | -| pylint | `# pylint: disable=` | line-level | Prefer refactor over suppression. | - -Every suppression MUST include a brief justification on the same line -(`# nosec B310 # scheme validated immediately above`). Unexplained suppressions will not -pass review. - -### Project-Wide Skip Configuration - -Systemic false positives are skipped at the config level, never with per-line comments. -The authoritative skip lists live in: - -- `.bandit` (YAML, with per-rule justification comments) — the canonical source. -- `pyproject.toml` `[tool.bandit]` — mirror for tooling that only reads `pyproject.toml`. - Keep both files in sync. - -When adding a new bandit skip: -1. Add it to `.bandit` with a `# B: ` comment. -2. Mirror it in `pyproject.toml` `[tool.bandit].skips`. -3. Verify locally: `py -m bandit -c pyproject.toml -r autopapertoppt/ sources/` must return - `No issues identified`. - -### Local CI Reproduction - -Before pushing, reproduce each engine locally so CI does not have to tell you: - -- **bandit**: `py -m bandit -c pyproject.toml -r autopapertoppt/ sources/` - (the `-c` flag is REQUIRED — without it, bandit ignores the skip config). -- **ruff**: `py -m ruff check .` -- **pytest**: `py -m pytest tests/` -- **search-mode smoke**: - `py -m autopapertoppt --query "diffusion models" --source arxiv --max 3 - --out ./smoke/` — confirm `.pptx`, `.xlsx`, `.bib` produced (the new search - default). -- **single-paper smoke**: - `py -m autopapertoppt --paper "https://arxiv.org/abs/1706.03762" --out - ./smoke/single/` — confirm `.pptx` + `.bib` only (single-paper default). -- **deck-overflow smoke** (when touching pptx/i18n): - inspect every shape's wrapped-text height ≤ its box AND ≤ 7.05" footer - guard. See `scripts/regen_ieee_thesis_style.py` for the inspection pattern. - -### Environment - -- Python 3.12+ (developed against 3.14) in the project-local `.venv/`. Activate - with `.venv\Scripts\Activate.ps1` (PowerShell) or `.venv\Scripts\activate.bat` - (cmd) before running `py -m ...` commands, OR call the venv interpreter - directly: `.venv\Scripts\python.exe -m pytest tests/`. -- Required runtime deps: `httpx`, `pydantic`, `pydantic-settings`, `defusedxml`, - `python-pptx`, `openpyxl`, `bibtexparser`, `beautifulsoup4`, `lxml`, - `markdown-it-py`. -- Optional extras (declared in `pyproject.toml`): - - `[intelligence]` — `pypdf` + `anthropic` for PDF extraction + `--enrich`. - - `[mcp]` — the `mcp` SDK for running / registering the MCP server. - - `[web]` — reserved for the future FastAPI / Streamlit UI. - - `[dev]` — all of the above + `pytest*`, `ruff`, `bandit`. - -### Env vars - -| Variable | Used by | Purpose | -|---|---|---| -| `ANTHROPIC_API_KEY` | `--enrich` Python path | LLM auth. **NOT** needed for the LLM-as-agent path over MCP. | -| `AUTOPAPERTOPPT_LLM_MODEL` | `--enrich` | Override the default `claude-opus-4-7`. | -| `AUTOPAPERTOPPT_S2_API_KEY` | Semantic Scholar plugin | Higher rate limit on `api.semanticscholar.org`. Optional. | -| `AUTOPAPERTOPPT_NCBI_API_KEY` | PubMed plugin | Raises NCBI's anonymous limit (3/s) to 10/s. Optional. | -| `AUTOPAPERTOPPT_CONTACT_EMAIL` | PubMed (`tool` / `email`), ACM/Crossref (`mailto`) | Puts Crossref in the polite pool. | -| `AUTOPAPERTOPPT_ENABLE_IEEE_SCRAPING` | IEEE plugin (scraping path) | Must be `=1`. IEEE Xplore ToS-grey. Not needed when `AUTOPAPERTOPPT_IEEE_API_KEY` is set. | -| `AUTOPAPERTOPPT_IEEE_API_KEY` | IEEE plugin (API path) | Switches the IEEE plugin to the official Xplore API (`ieeexploreapi.ieee.org`). Surfaces `pdf_url` for papers in the key's subscription scope. Apply at https://developer.ieee.org/. | -| `AUTOPAPERTOPPT_CROSSREF_PLUS_TOKEN` | ACM / Crossref plugin | Crossref Plus subscriber token; attached as `Crossref-Plus-API-Token: Bearer …`. Raises rate limits + cache freshness. | -| `AUTOPAPERTOPPT_SPRINGER_API_KEY` | Springer plugin | Free key from https://dev.springernature.com/. Required — the Springer plugin raises `ConfigError` without it. Covers Nature, Scientific Reports, Lecture Notes in CS. | -| `AUTOPAPERTOPPT_PDF_COOKIES_FILE` | PDF downloader | Path to a Netscape-format `cookies.txt` file. Cookies whose domain matches a PDF URL's host are attached on the request. Off by default. Use when publishers return 403 to anonymous requests for paywalled PDFs you have institutional access to. **You are responsible for compliance with each publisher's terms of service.** A startup warning fires when the env var is loaded. | -| `AUTOPAPERTOPPT_ENABLE_SCHOLAR_SCRAPING` | Scholar plugin | Must be `=1`. Google Scholar terms forbid scraping; off by default. | -| `AUTOPAPERTOPPT_LOG_LEVEL` | logger | `INFO` default; set `DEBUG` for verbose tracing. | +- NEVER add `Co-Authored-By` lines. +- NEVER mention "Claude", "Claude Code", "AI-generated", "GPT", "Copilot", or any + AI tool / model name anywhere — commit messages, PR titles, PR descriptions, + code comments, documentation. + +## IEEE / Publisher CDN: Browser Automation Is Mandatory (HARD RULE) + +**Before triggering ANY search that involves paywalled publishers +(IEEE / ACM / Springer / etc.), the LLM in this session MUST confirm +the user's VPN / institutional access status first** — either by +recalling a recent statement, or by asking via `AskUserQuestion` +("Do you have VPN for IEEE / ACM / Springer for this topic?"). +Without VPN, IEEE returns abstract-only / 403 for the PDF stage and +the per-paper download fails. When the user says no VPN, restrict +the search to `arxiv,openalex,pubmed,crossref,dblp,openaire,scholar` +— that is, **skip only `ieee`**. Google Scholar is publicly +accessible and stays in the mix even without VPN (Chrome still boots +for it because of captcha resilience, but the search itself works). +This gate applies BEFORE running `python -m autopapertoppt -q …`, +before `scripts/llm_driven_search.py`, and before any +`scripts/llm_download_*pdf*.py` invocation. + +IEEE search, IEEE document fetch, Google Scholar search, and any paywalled-PDF +download from publisher CDNs (ieeexplore.ieee.org, dl.acm.org, link.springer.com, +sciencedirect.com, wiley/oup/nature/science/…) MUST go through **visible Chrome**. +Two paths exist: + +1. **Python pipeline** — IEEE / Scholar plugins call their own `webrunner_backend` + from inside `asyncio.gather`. Used by the CLI in unattended mode. +2. **LLM-as-agent** — the LLM in a Claude Code session drives Chrome itself via + Bash + `autopapertoppt.fetchers.webrunner_browser.make_driver()`. Reference: + `scripts/llm_driven_search.py` + `scripts/llm_parse_results.py`. The + `mcp__webrunner__*` server registered for this project only exposes static + helpers (lint / translate / score) — it does NOT expose + `webrunner_run_actions` or any other browser-driving tool, so the LLM cannot + skip the Bash + Selenium step. + +The httpx branch in those plugins is a CI safety net for no-Chrome environments; +on a user machine with VPN, silent fall-through to httpx is a bug. **Never +suppress the visible window** (`--headless`, etc.). If you don't see a Chrome +window open during an IEEE / Scholar / paywalled-PDF step, the path is broken +— surface it, don't trust the results. Full rule + audit checklist: +`compliance-auditor` subagent. + +## Where the detailed rules live + +| Topic | Subagent (in `.claude/agents/`) | +|---|---| +| Design patterns, SOLID, performance, async, unit tests, full linter rule set | `code-quality-reviewer` | +| Core-vs-source-plugin boundary, network safety, browser-automation hard rule, path safety, suppression conventions, bandit skip config | `compliance-auditor` | +| pptx exporter geometry, rendering tiers, truncation caps, semantic shape names, i18n, LLM-as-agent vs Python pipeline | `slide-deck-rules` | +| Env vars + Python / `.venv` toolchain reference | `env-vars` | +| Definition-of-Done gate runner | `dod-verify` | +| LLM-as-agent thesis-style authoring (PDF → rich PaperSummary) | `paper-summary-author` | +| URL-fabrication / off-topic audits after authoring | `post-author-audit` | +| Slide-overflow regression check | `slide-overflow-check` | diff --git a/README.md b/README.md index ee347b5..cd293e8 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **Languages**: **English** · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) +> **Languages**: **English** · [繁體中文](readmes/README.zh-TW.md) · [简体中文](readmes/README.zh-CN.md) · [日本語](readmes/README.ja.md) · [Español](readmes/README.es.md) · [Français](readmes/README.fr.md) · [Deutsch](readmes/README.de.md) · [한국어](readmes/README.ko.md) · [Português](readmes/README.pt.md) · [Русский](readmes/README.ru.md) · [Italiano](readmes/README.it.md) · [Tiếng Việt](readmes/README.vi.md) · [हिन्दी](readmes/README.hi.md) · [Bahasa Indonesia](readmes/README.id.md) > **Documentation**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) A keyword-driven paper search assistant that fetches results from arXiv, diff --git a/autopapertoppt/cli.py b/autopapertoppt/cli.py index c5a0b99..62fe997 100644 --- a/autopapertoppt/cli.py +++ b/autopapertoppt/cli.py @@ -243,18 +243,31 @@ def build_parser() -> argparse.ArgumentParser: ) parser.set_defaults(download_pdf=True) parser.add_argument( - "--all-venues", + "--top-tier-only", dest="top_tier_only", + action="store_true", + help=( + "Restrict results to papers from arXiv or from the curated " + "top-tier CS venue whitelist (S&P, CCS, NDSS, USENIX Security, " + "NeurIPS, ICML, ICSE, SIGMOD, SIGCOMM, CHI, etc.). Off by " + "default — most IEEE / ACM workshop papers live outside the " + "whitelist and would be filtered out otherwise." + ), + ) + parser.set_defaults(top_tier_only=False) + parser.add_argument( + "--no-oa-resolve", + dest="resolve_oa", action="store_false", help=( - "Disable the top-tier CS venue filter. By default the search " - "keeps only papers from arXiv or from a curated whitelist of " - "top-tier conferences / journals (S&P, CCS, NDSS, USENIX " - "Security, NeurIPS, ICML, ICSE, SIGMOD, SIGCOMM, CHI, etc.). " - "Pass --all-venues to keep every result regardless of venue." + "Skip the open-access PDF resolver step that runs after dedup. " + "By default the pipeline looks up every paper without pdf_url " + "in Unpaywall (needs AUTOPAPERTOPPT_CONTACT_EMAIL) and falls " + "back to an arXiv title search — typical lift of 40-70 percent " + "for IEEE / ACM / Springer / Elsevier paywalled papers." ), ) - parser.set_defaults(top_tier_only=True) + parser.set_defaults(resolve_oa=True) parser.add_argument( "--paywall-threshold", type=float, @@ -540,7 +553,7 @@ async def _collect(args: argparse.Namespace): top_tier_only=args.top_tier_only, ) _LOG.info("Running search: %s across %s", keywords, ", ".join(sources)) - return await run_search(query) + return await run_search(query, resolve_oa=args.resolve_oa) def _resolve_enrich_mode(args: argparse.Namespace) -> str: diff --git a/autopapertoppt/core/constants.py b/autopapertoppt/core/constants.py index 5f766f8..8d51d44 100644 --- a/autopapertoppt/core/constants.py +++ b/autopapertoppt/core/constants.py @@ -48,15 +48,13 @@ SOURCE_SPRINGER, ) ALL_SOURCES: tuple[str, ...] = CORE_SOURCES + PLUGIN_SOURCES -# Sources tried by default when --source is not given. Mixes the open / free -# endpoints. OpenAlex sits in the default mix because it surfaces direct -# ``pdf_url`` for many papers whose publisher pages are paywalled (IEEE, -# ACM, Elsevier), via author / institutional OA mirrors. DBLP + Crossref + -# OpenAIRE join the default mix because they need no API key and broaden -# coverage to CS bibliography (DBLP), every Crossref-indexed publisher, and -# European OA repositories (OpenAIRE). Opt-in plugins (ieee, scholar, -# springer) join only when their env var is set; the pipeline skips them -# silently otherwise so this list stays safe as a default. +# Sources tried by default when --source is not given. The full source mix +# is on by default for maximum coverage. ieee and scholar are now also +# default-on (their scrape paths gated by AUTOPAPERTOPPT_DISABLE_*_SCRAPING +# opt-out env vars instead of the previous opt-in vars). springer still +# raises ConfigError at construction without an API key, so the pipeline +# silently skips it — leaving it in the list is harmless and keeps it +# easy to enable by setting AUTOPAPERTOPPT_SPRINGER_API_KEY. DEFAULT_SOURCES: tuple[str, ...] = ( SOURCE_ARXIV, SOURCE_SEMANTIC_SCHOLAR, @@ -68,6 +66,7 @@ SOURCE_CROSSREF, SOURCE_OPENAIRE, SOURCE_SPRINGER, + SOURCE_SCHOLAR, ) EXPORT_BIBTEX: str = "bib" diff --git a/autopapertoppt/core/oa_resolver.py b/autopapertoppt/core/oa_resolver.py new file mode 100644 index 0000000..44619ef --- /dev/null +++ b/autopapertoppt/core/oa_resolver.py @@ -0,0 +1,394 @@ +"""Post-dedup PDF resolver — fill missing pdf_url from open-access aggregators. + +Why this exists +--------------- +Most IEEE / ACM / Springer / Elsevier papers come back from their +respective source plugins with ``pdf_url=None`` because the publisher +sites are paywalled even when the paper itself is open access. The OA +copy almost always exists somewhere else — the author's institutional +repository, an arXiv preprint, ResearchGate, etc. + +This module runs after dedup and tries four strategies in order for +every paper that still lacks a pdf_url: + +1. **arXiv-ID direct**. If the paper carries ``arxiv_id`` (set by + the openalex / pubmed / crossref / semantic_scholar parsers when + the upstream identified an arXiv preprint), turn it into + ``https://arxiv.org/pdf/{arxiv_id}.pdf`` directly. Zero network + round-trip; highest precision; fastest. + +2. **Unpaywall** (https://api.unpaywall.org/v2). Free, no API key, + ~50M papers. Needs ``AUTOPAPERTOPPT_CONTACT_EMAIL`` for politeness + (skipped silently when unset). + +3. **Semantic Scholar OA index** (https://api.semanticscholar.org/graph/v1/paper/DOI:{doi}). + S2's ``openAccessPdf`` index is partially disjoint from Unpaywall; + when one misses the other often hits. Free, no API key required + (rate-limited to ~1 req/s anonymous; an + ``AUTOPAPERTOPPT_S2_API_KEY`` raises that). + +4. **CORE.ac.uk** (https://api.core.ac.uk/v3/search/works). Aggregator + of 200M+ OA repository items — institutional repos, regional + preprint servers, OA journals. Needs ``AUTOPAPERTOPPT_CORE_API_KEY`` + (free at https://core.ac.uk/services/api); skipped silently when + unset. + +5. **arXiv title search**. For papers without DOI / arxiv_id, search + arXiv with the paper's title. Exact-match on the normalised title + (alphanumeric + lowercase) so loosely-similar titles do not get + adopted by accident. + +Every lookup is best-effort: any failure logs at DEBUG and the +paper passes through unchanged. The resolver never raises. +""" + +from __future__ import annotations + +import asyncio +import dataclasses +import os +from typing import Any + +import httpx + +from autopapertoppt.core.exceptions import FetchError +from autopapertoppt.core.models import Paper, PaperCollection, Query +from autopapertoppt.fetchers.http import get_client +from autopapertoppt.utils.logging import get_logger + +_LOG = get_logger(__name__) + +_UNPAYWALL_ENDPOINT = "https://api.unpaywall.org/v2" +_UNPAYWALL_SOURCE = "unpaywall" +_S2_ENDPOINT = "https://api.semanticscholar.org/graph/v1/paper" +_S2_SOURCE = "semantic_scholar_oa" +_CORE_ENDPOINT = "https://api.core.ac.uk/v3/search/works" +_CORE_SOURCE = "core_ac_uk" +_LOOKUP_TIMEOUT_SECONDS = 10.0 +_CONCURRENCY = 5 + +# One-shot warnings so we don't spam logs for every paper in a large run. +_email_warning_emitted = False +_core_warning_emitted = False + +#: In-process cache for S2 OA lookups, keyed by DOI. Prevents the +#: resolver from re-hitting S2 for the same paper across multiple +#: searches in one CLI run. +_S2_CACHE: dict[str, str | None] = {} + + +async def resolve_oa_pdfs(collection: PaperCollection) -> PaperCollection: + """Try to fill ``pdf_url`` for every paper currently missing one. + + Returns a new ``PaperCollection`` with the same query and same + paper count. Papers that already have ``pdf_url`` pass through + unchanged. + """ + missing = sum(1 for p in collection.papers if not p.pdf_url) + if missing == 0: + return collection + + _LOG.info( + "OA resolver: looking up %d / %d papers without pdf_url", + missing, + len(collection.papers), + ) + + semaphore = asyncio.Semaphore(_CONCURRENCY) + resolved = await asyncio.gather( + *(_resolve_one(paper, semaphore) for paper in collection.papers) + ) + found = sum( + 1 + for old, new in zip(collection.papers, resolved, strict=True) + if not old.pdf_url and new.pdf_url + ) + if found: + _LOG.info( + "OA resolver: filled %d / %d missing pdf_url (Unpaywall + arXiv)", + found, + missing, + ) + return PaperCollection(query=collection.query, papers=tuple(resolved)) + + +#: DOI-keyed OA lookup strategies, tried in order until one returns a URL. +_DOI_STRATEGIES = ( + ("Unpaywall", lambda doi: _query_unpaywall(doi)), + ("S2 OA", lambda doi: _query_semantic_scholar(doi)), + ("CORE", lambda doi: _query_core(doi)), +) + + +async def _resolve_one(paper: Paper, semaphore: asyncio.Semaphore) -> Paper: + if paper.pdf_url: + return paper + async with semaphore: + pdf = await _try_all_strategies(paper) + if pdf: + return dataclasses.replace(paper, pdf_url=pdf) + return paper + + +async def _try_all_strategies(paper: Paper) -> str | None: + """Run every OA strategy in priority order, returning the first hit.""" + key = paper.bibtex_key() + # 1. arXiv-ID direct — no round-trip, highest precision. + if paper.arxiv_id: + pdf = _arxiv_id_to_pdf(paper.arxiv_id) + if pdf: + _LOG.debug("arxiv_id direct hit for %s: %s", key, pdf) + return pdf + # 2-4. DOI-keyed external aggregators. + if paper.doi: + for label, query in _DOI_STRATEGIES: + pdf = await query(paper.doi) + if pdf: + _LOG.debug("%s hit for %s: %s", label, key, pdf) + return pdf + # 5. arXiv title search — last resort for DOI-less papers. + pdf = await _query_arxiv_title(paper) + if pdf: + _LOG.debug("arXiv title hit for %s: %s", key, pdf) + return pdf + return None + + +def _arxiv_id_to_pdf(arxiv_id: str) -> str | None: + """Derive the canonical arXiv PDF URL from an arXiv ID. + + Strips any trailing ``v`` version suffix because arXiv resolves + bare IDs to the latest version automatically. + """ + cleaned = arxiv_id.strip() + if not cleaned: + return None + # 1706.03762v2 → 1706.03762; cs.LG/0001001v1 → cs.LG/0001001 + if "v" in cleaned: + base, _, tail = cleaned.rpartition("v") + if tail.isdigit() and base: + cleaned = base + return f"https://arxiv.org/pdf/{cleaned}.pdf" + + +async def _query_unpaywall(doi: str) -> str | None: + """Look up a DOI in Unpaywall; return the best OA PDF URL or None.""" + email = os.environ.get("AUTOPAPERTOPPT_CONTACT_EMAIL", "").strip() + if not email: + _warn_once_about_email() + return None + client = await get_client(_UNPAYWALL_SOURCE) + try: + response = await asyncio.wait_for( + client.get( + f"{_UNPAYWALL_ENDPOINT}/{doi}", + params={"email": email}, + ), + timeout=_LOOKUP_TIMEOUT_SECONDS, + ) + except (TimeoutError, httpx.HTTPError, FetchError) as err: + _LOG.debug("Unpaywall lookup failed for %s: %s", doi, err) + return None + if response.status_code == 404: + return None # not indexed + if response.status_code != 200: + _LOG.debug( + "Unpaywall returned %s for %s: %s", + response.status_code, doi, response.text[:128], + ) + return None + try: + data: dict[str, Any] = response.json() + except ValueError: + return None + best_oa = data.get("best_oa_location") or {} + candidate = (best_oa.get("url_for_pdf") or "").strip() + if candidate.startswith("https://"): + return candidate + return None + + +async def _query_semantic_scholar(doi: str) -> str | None: + """Look up a DOI in Semantic Scholar's OA index. + + Honours ``AUTOPAPERTOPPT_S2_API_KEY`` (sent as the ``x-api-key`` + header) for the higher rate limit. Without the key the anonymous + tier (~1 req/s) is fragile under burst; the resolver is rate-limit + tolerant (it falls back to other strategies on any non-200) but + you'll see a lot more 429s. + """ + if doi in _S2_CACHE: + return _S2_CACHE[doi] + client = await get_client(_S2_SOURCE) + headers: dict[str, str] = {} + api_key = os.environ.get("AUTOPAPERTOPPT_S2_API_KEY", "").strip() + if api_key: + headers["x-api-key"] = api_key + try: + response = await asyncio.wait_for( + client.get( + f"{_S2_ENDPOINT}/DOI:{doi}", + params={"fields": "openAccessPdf"}, + headers=headers or None, + ), + timeout=_LOOKUP_TIMEOUT_SECONDS, + ) + except (TimeoutError, httpx.HTTPError, FetchError) as err: + _LOG.debug("S2 OA lookup failed for %s: %s", doi, err) + return None + if response.status_code == 404: + _S2_CACHE[doi] = None + return None + if response.status_code == 429: + # Anonymous rate-limit. Don't cache — try again in a future + # resolver pass (e.g. if user retries the search after setting + # AUTOPAPERTOPPT_S2_API_KEY). + _LOG.debug("S2 rate-limited for %s; not caching", doi) + return None + if response.status_code != 200: + _LOG.debug( + "S2 returned %s for %s: %s", + response.status_code, doi, response.text[:128], + ) + return None + try: + data: dict[str, Any] = response.json() + except ValueError: + return None + pdf_obj = data.get("openAccessPdf") or {} + candidate = (pdf_obj.get("url") or "").strip() if isinstance(pdf_obj, dict) else "" + if candidate.startswith("https://"): + _S2_CACHE[doi] = candidate + return candidate + _S2_CACHE[doi] = None + return None + + +async def _query_core(doi: str) -> str | None: + """Look up a DOI on CORE.ac.uk for OA repository copies.""" + api_key = os.environ.get("AUTOPAPERTOPPT_CORE_API_KEY", "").strip() + if not api_key: + _warn_once_about_core() + return None + client = await get_client(_CORE_SOURCE) + try: + response = await asyncio.wait_for( + client.get( + _CORE_ENDPOINT, + params={"q": f'doi:"{doi}"', "limit": "1"}, + headers={"Authorization": f"Bearer {api_key}"}, + ), + timeout=_LOOKUP_TIMEOUT_SECONDS, + ) + except (TimeoutError, httpx.HTTPError, FetchError) as err: + _LOG.debug("CORE lookup failed for %s: %s", doi, err) + return None + if response.status_code != 200: + _LOG.debug( + "CORE returned %s for %s: %s", + response.status_code, doi, response.text[:128], + ) + return None + try: + data: dict[str, Any] = response.json() + except ValueError: + return None + results = data.get("results") or [] + if not results: + return None + first = results[0] + # CORE's `downloadUrl` is the direct PDF; fall back to `fullTextLinks` + # otherwise. + candidate = (first.get("downloadUrl") or "").strip() + if candidate.startswith("https://"): + return candidate + for link in first.get("fullTextLinks") or []: + url = (link.get("url") or "").strip() if isinstance(link, dict) else "" + if url.startswith("https://"): + return url + return None + + +async def _query_arxiv_title(paper: Paper) -> str | None: + """Search arXiv by title; return the matching paper's PDF URL or None. + + Match is exact on the normalised title (alphanumeric + lowercase) + so a "transformer" paper doesn't accidentally claim someone else's + "transformer architecture for X" preprint. + """ + if not paper.title: + return None + # Skip the round-trip if the paper is already from arXiv — its + # plugin would have populated pdf_url at parse time if a PDF + # existed. + if paper.source == "arxiv": + return None + try: + from autopapertoppt.fetchers.base import load_fetcher + except ImportError: + return None + try: + fetcher = load_fetcher("arxiv") + except Exception: # noqa: BLE001 — load failures must not break the resolver + return None + + # arXiv's API supports field-restricted queries; ti:"" looks + # only at the title field. Pull the top 3 in case the first is a + # later version of a different paper with similar words. + query = Query( + keywords=f'ti:"{paper.title}"', + sources=("arxiv",), + max_results=3, + ) + try: + results = await fetcher.search(query) + except Exception as err: # noqa: BLE001 — best-effort + _LOG.debug("arXiv title search failed for %r: %s", paper.title, err) + return None + + target = _normalise_title(paper.title) + for candidate in results: + if ( + _normalise_title(candidate.title) == target + and candidate.pdf_url + and candidate.pdf_url.startswith("https://") + ): + return candidate.pdf_url + return None + + +def _normalise_title(text: str) -> str: + """Lowercase + drop non-alphanumeric for fuzzy title comparison.""" + return "".join(c.lower() for c in text if c.isalnum()) + + +def _warn_once_about_email() -> None: + """Log a single WARNING line when CONTACT_EMAIL is unset.""" + global _email_warning_emitted # noqa: PLW0603 — intentional one-shot flag + if _email_warning_emitted: + return + _email_warning_emitted = True + _LOG.warning( + "OA resolver: AUTOPAPERTOPPT_CONTACT_EMAIL is not set; " + "Unpaywall lookups (the biggest PDF coverage win for IEEE / " + "ACM / Springer / Elsevier papers) will be skipped. Set the " + "env var to your email to enable them." + ) + + +def _warn_once_about_core() -> None: + """Log a single WARNING line when CORE_API_KEY is unset. + + CORE is an optional layer on top of Unpaywall + S2 + arXiv; the + warning is INFO-level rather than WARNING because most users + will be fine without it. + """ + global _core_warning_emitted # noqa: PLW0603 — intentional one-shot flag + if _core_warning_emitted: + return + _core_warning_emitted = True + _LOG.info( + "OA resolver: AUTOPAPERTOPPT_CORE_API_KEY is not set; CORE.ac.uk " + "lookups (institutional / regional OA repos) will be skipped. " + "Get a free key at https://core.ac.uk/services/api to enable." + ) diff --git a/autopapertoppt/core/pdf_download.py b/autopapertoppt/core/pdf_download.py index 2a5edc4..a7431ba 100644 --- a/autopapertoppt/core/pdf_download.py +++ b/autopapertoppt/core/pdf_download.py @@ -76,6 +76,20 @@ async def _download_one(paper: Paper, pdf_dir: Path) -> PdfDownloadResult: if target.exists() and target.stat().st_size > 0: _LOG.info("pdf already on disk for %s: %s", key, target) return PdfDownloadResult(paper_key=key, path=target, skipped_reason=None) + # For paywalled publisher CDNs (IEEE, ACM, Springer, Elsevier, ...) + # httpx-style requests reliably 403. Route those through WebRunner + # (real visible Chrome) so the session cookie / TLS handshake / JS + # fingerprint match what the publisher expects. + from autopapertoppt.fetchers import webrunner_pdf + + if webrunner_pdf.is_available() and webrunner_pdf.should_use_webrunner(paper.pdf_url): + _LOG.info("pdf via WebRunner for %s: %s", key, paper.pdf_url) + ok = await webrunner_pdf.download_via_browser(paper.pdf_url, target) + if ok: + return PdfDownloadResult(paper_key=key, path=target, skipped_reason=None) + _LOG.info( + "pdf WebRunner failed for %s; falling back to httpx", key, + ) return await _fetch_and_validate(paper, target, key) diff --git a/autopapertoppt/core/pipeline.py b/autopapertoppt/core/pipeline.py index 67713dc..85a4fcd 100644 --- a/autopapertoppt/core/pipeline.py +++ b/autopapertoppt/core/pipeline.py @@ -23,6 +23,7 @@ ) from autopapertoppt.core.identifiers import PaperIdentifier from autopapertoppt.core.models import Paper, PaperCollection, Query +from autopapertoppt.core.oa_resolver import resolve_oa_pdfs from autopapertoppt.core.ranking import rank from autopapertoppt.core.top_venues import is_top_tier from autopapertoppt.fetchers.base import load_fetcher @@ -31,11 +32,19 @@ _LOG = get_logger(__name__) -async def run_search(query: Query) -> PaperCollection: +async def run_search( + query: Query, *, resolve_oa: bool = True +) -> PaperCollection: """Run `query` across its sources concurrently and produce a collection. Source plugins that fail to load (e.g. an opt-in plugin whose env var is unset) are skipped with a warning so the rest of the mix still runs. + + ``resolve_oa`` (default True) runs the OA PDF resolver after dedup + + rank + top-tier filter so papers whose source returned no ``pdf_url`` + (typical for IEEE / ACM / Springer / Elsevier) get a chance to pick + up an open-access mirror from Unpaywall or an arXiv preprint. + Pass ``False`` from tests or CLI flags that want raw source output. """ fetchers = [ loaded @@ -58,7 +67,12 @@ async def run_search(query: Query) -> PaperCollection: _LOG.info( "top-tier filter kept %d / %d papers", len(ordered), before ) - return PaperCollection(query=query, papers=tuple(ordered[: query.max_results])) + collection = PaperCollection( + query=query, papers=tuple(ordered[: query.max_results]) + ) + if resolve_oa: + collection = await resolve_oa_pdfs(collection) + return collection def _load_fetcher_safe(name: str): diff --git a/autopapertoppt/fetchers/base.py b/autopapertoppt/fetchers/base.py index 1f9832a..fdef4ad 100644 --- a/autopapertoppt/fetchers/base.py +++ b/autopapertoppt/fetchers/base.py @@ -24,7 +24,13 @@ class FetcherConfig: rate_limit: RateLimit requires_api_key: bool = False enabled_by_default: bool = True + # Env var the plugin checks to enable itself (e.g. Springer needs an + # API key; without it the plugin raises ConfigError at construction). opt_in_env_var: str | None = None + # Env var the plugin checks to disable itself when it is otherwise + # default-ON. Used by IEEE / Scholar — both scrape paths are + # default-on but the user can flip them off via this env var. + opt_out_env_var: str | None = None class Fetcher(ABC): diff --git a/autopapertoppt/fetchers/webrunner_browser.py b/autopapertoppt/fetchers/webrunner_browser.py new file mode 100644 index 0000000..bb34336 --- /dev/null +++ b/autopapertoppt/fetchers/webrunner_browser.py @@ -0,0 +1,144 @@ +"""Shared raw-Selenium helpers for WebRunner-based source plugins. + +Why not je_web_runner directly +------------------------------ +``je_web_runner.webdriver_wrapper_instance`` is a module-level +singleton. When the pipeline runs sources concurrently +(``asyncio.gather`` fans out Scholar, IEEE, etc. simultaneously), +the Scholar backend and the IEEE backend both call +``set_driver(...)`` against the SAME singleton — they fight over it, +one Chrome becomes orphaned, the other's ``execute_async_script`` / +``page_source`` reads a window in the wrong state. The symptom is +silent: no exception, no log, just a hung Chrome window stuck on a +home page and an empty result set from the affected source. + +This module sidesteps the singleton by spinning up a fresh +``selenium.webdriver.Chrome`` per call. Each WebRunner-backed search +owns its driver, never shares state, and quits cleanly when done. +""" + +from __future__ import annotations + +import os +import time +from typing import Any + +from autopapertoppt.utils.logging import get_logger + +_LOG = get_logger(__name__) + +_DISABLE_ENV = "AUTOPAPERTOPPT_DISABLE_WEBRUNNER" +_PROFILE_DIR_ENV = "AUTOPAPERTOPPT_CHROME_PROFILE_DIR" + +#: URL fragments and body markers that indicate the page is a captcha +#: / 'unusual traffic' interstitial instead of the expected content. +#: Combined Google + IEEE patterns; safe to grep both. +_CAPTCHA_URL_FRAGMENTS: tuple[str, ...] = ( + "/sorry/", + "/captcha", + "/recaptcha", +) +_CAPTCHA_BODY_MARKERS: tuple[str, ...] = ( + "Our systems have detected unusual traffic", + 'id="captcha-form"', + "g-recaptcha", + "Please show you're not a robot", + "Please verify you're a human", + "Verify you're not a robot", + "Access blocked", +) + + +def is_available() -> bool: + """True when ``selenium`` is importable AND ``AUTOPAPERTOPPT_DISABLE_WEBRUNNER`` + is not set. + """ + if os.environ.get(_DISABLE_ENV) == "1": + return False + try: + import selenium # noqa: F401 + except ImportError: + return False + return True + + +def make_driver(*, download_dir: str | None = None) -> Any: + """Boot a fresh visible Chrome with anti-detection options. + + Returns a ``selenium.webdriver.Chrome`` instance the caller is + responsible for closing (``.quit()``). When ``download_dir`` is + provided, Chrome is configured to save PDFs straight to disk + instead of opening the built-in viewer. + """ + from selenium import webdriver + from selenium.webdriver.chrome.options import Options + + options = Options() + options.add_argument("--disable-blink-features=AutomationControlled") + options.add_argument("--lang=en-US") + options.add_argument("--disable-gpu") + options.add_argument("--no-sandbox") + options.add_argument("--window-size=1280,720") + profile_dir = os.environ.get(_PROFILE_DIR_ENV, "").strip() + if profile_dir: + options.add_argument(f"--user-data-dir={profile_dir}") + if download_dir: + prefs = { + "download.default_directory": download_dir, + "download.prompt_for_download": False, + "download.directory_upgrade": True, + "plugins.always_open_pdf_externally": True, + "safebrowsing.enabled": False, + } + options.add_experimental_option("prefs", prefs) + return webdriver.Chrome(options=options) + + +def is_captcha_page(driver: Any) -> bool: + """True when the driver is currently on a captcha / blocked page.""" + try: + current_url = driver.current_url or "" + body = driver.page_source or "" + except Exception: # noqa: BLE001 — best-effort detection + return False + if any(fragment in current_url for fragment in _CAPTCHA_URL_FRAGMENTS): + return True + head = body[:8192] + return any(marker in head for marker in _CAPTCHA_BODY_MARKERS) + + +def wait_for_captcha_solved( + driver: Any, + *, + max_wait_seconds: float = 300.0, + poll_interval: float = 2.0, +) -> bool: + """Wait for the user to solve a captcha visible in the Chrome window. + + If the current page is NOT a captcha, returns True immediately. + Otherwise polls every ``poll_interval`` seconds until either the + captcha state clears (user solved it; URL changes back to the + real page) or ``max_wait_seconds`` elapses. + + Returns True when the captcha cleared, False when the max wait + elapsed without resolution. Never raises. + """ + if not is_captcha_page(driver): + return True + try: + starting_url = driver.current_url or "" + except Exception: # noqa: BLE001 + starting_url = "" + _LOG.warning( + "Captcha / 'unusual traffic' page detected at %s. Solve it in " + "the visible Chrome window — waiting up to %.0fs.", + starting_url, max_wait_seconds, + ) + deadline = time.monotonic() + max_wait_seconds + while time.monotonic() < deadline: + time.sleep(poll_interval) + if not is_captcha_page(driver): + _LOG.info("Captcha cleared — continuing.") + return True + _LOG.warning("Captcha not solved within timeout; giving up on this source.") + return False diff --git a/autopapertoppt/fetchers/webrunner_pdf.py b/autopapertoppt/fetchers/webrunner_pdf.py new file mode 100644 index 0000000..9e72fbe --- /dev/null +++ b/autopapertoppt/fetchers/webrunner_pdf.py @@ -0,0 +1,158 @@ +"""PDF download via WebRunner (real visible Chrome browser). + +Why +--- +Publisher PDF CDNs (IEEE Xplore, ACM Digital Library, Springer, Elsevier, +Wiley, Taylor & Francis, etc.) return 403 to httpx-style requests even +with browser headers + Referer + cookies. They fingerprint the TLS +handshake and the JavaScript engine to require a real Chrome. + +This module routes PDF downloads for paywalled publisher domains +through a real visible Chrome instance configured to save PDFs +directly to disk (instead of opening the built-in PDF viewer). The +profile dir env var the rest of WebRunner uses is honoured here too, +so institutional auth cookies surface paywalled subscription PDFs +the same as they would in a normal browser session. + +The actual Selenium calls run inside ``asyncio.to_thread`` so the +download doesn't block the pipeline's event loop while Chrome boots ++ waits for the file to appear (5-30s per PDF). +""" + +from __future__ import annotations + +import asyncio +import contextlib +import os +import shutil +import tempfile +import time +from pathlib import Path +from urllib.parse import urlparse + +from autopapertoppt.utils.logging import get_logger + +_LOG = get_logger(__name__) + +_DISABLE_ENV = "AUTOPAPERTOPPT_DISABLE_WEBRUNNER" +_PROFILE_DIR_ENV = "AUTOPAPERTOPPT_CHROME_PROFILE_DIR" +#: Per-PDF wall-clock cap. Generous to handle 50-page Elsevier PDFs on +#: slow connections; Chrome boot + page-load is usually the bigger +#: fraction of this budget. +_DOWNLOAD_TIMEOUT_SECONDS = 60.0 +_DOWNLOAD_POLL_INTERVAL = 0.5 + +#: Publisher CDN hosts where httpx-style PDF GETs reliably 403. +#: Anything resolved on these hosts is routed through WebRunner. +#: Subdomain matching: `endswith` on the hostname so +#: e.g. ``onlinelibrary.wiley.com`` matches the broader ``wiley.com`` +#: entry. +_PAYWALLED_SUFFIXES: tuple[str, ...] = ( + "ieeexplore.ieee.org", + "ieee.org", + "dl.acm.org", + "acm.org", + "link.springer.com", + "springer.com", + "sciencedirect.com", + "elsevier.com", + "onlinelibrary.wiley.com", + "wiley.com", + "tandfonline.com", + "academic.oup.com", + "oup.com", + "nature.com", + "science.org", + "asme.org", + "asce.org", + "ascelibrary.org", +) + + +def is_available() -> bool: + """True when je_web_runner is importable AND not explicitly disabled.""" + if os.environ.get(_DISABLE_ENV) == "1": + return False + try: + import selenium # noqa: F401 + except ImportError: + return False + return True + + +def should_use_webrunner(url: str) -> bool: + """True when the URL's host is a known paywalled publisher CDN.""" + host = (urlparse(url).hostname or "").lower() + if not host: + return False + return any(host.endswith(suffix) for suffix in _PAYWALLED_SUFFIXES) + + +async def download_via_browser(url: str, target: Path) -> bool: + """Drive Chrome to download a PDF, copy it to ``target``. + + Returns True on success (target file written and ≥ 4 bytes starting + with ``%PDF``), False on any failure. Never raises — callers fall + back to the httpx path on False. + """ + return await asyncio.to_thread(_download_sync, url, target) + + +def _download_sync(url: str, target: Path) -> bool: + """Boot Chrome → navigate to PDF URL → wait for file → copy to target.""" + from autopapertoppt.fetchers import webrunner_browser + + tmpdir = Path(tempfile.mkdtemp(prefix="autopapertoppt_pdf_")) + try: + try: + driver = webrunner_browser.make_driver(download_dir=str(tmpdir)) + except Exception as err: # noqa: BLE001 — Selenium raises many types + _LOG.warning("WebRunner PDF: cannot start Chrome: %s", err) + return False + try: + return _navigate_and_collect(driver, url, tmpdir, target) + finally: + with contextlib.suppress(Exception): + driver.quit() + finally: + shutil.rmtree(tmpdir, ignore_errors=True) + + +def _navigate_and_collect(driver, url: str, tmpdir: Path, target: Path) -> bool: + """Navigate to ``url``, poll ``tmpdir`` for a finished PDF, copy to target.""" + try: + driver.get(url) + except Exception as err: # noqa: BLE001 + _LOG.warning("WebRunner PDF: navigation failed for %s: %s", url, err) + return False + + deadline = time.monotonic() + _DOWNLOAD_TIMEOUT_SECONDS + while time.monotonic() < deadline: + partials = list(tmpdir.glob("*.crdownload")) + completed = [p for p in tmpdir.iterdir() if p.suffix.lower() == ".pdf"] + if completed and not partials: + return _persist_downloaded_pdf(completed[0], target) + time.sleep(_DOWNLOAD_POLL_INTERVAL) + _LOG.warning("WebRunner PDF: timed out waiting for %s", url) + return False + + +def _persist_downloaded_pdf(source: Path, target: Path) -> bool: + """Validate the magic bytes, copy to ``target``, return success.""" + try: + head = source.read_bytes()[:4] + except OSError as err: + _LOG.warning("WebRunner PDF: cannot read %s: %s", source, err) + return False + if not head.startswith(b"%PDF"): + _LOG.warning("WebRunner PDF: %s is not a PDF (head=%r)", source, head) + return False + target.parent.mkdir(parents=True, exist_ok=True) + try: + shutil.move(str(source), str(target)) + except OSError as err: + _LOG.warning( + "WebRunner PDF: cannot move %s -> %s: %s", source, target, err, + ) + return False + return True diff --git a/autopapertoppt/mcp/server.py b/autopapertoppt/mcp/server.py index 1550df5..b68b6f1 100644 --- a/autopapertoppt/mcp/server.py +++ b/autopapertoppt/mcp/server.py @@ -55,14 +55,20 @@ _LOG = get_logger(__name__) -# Plugins that refuse to load without an env var. Mirrors the in-fetcher -# ConfigError checks (ieee/__init__.py + springer/fetcher.py + scholar/...) -# so list_sources can report enablement without round-tripping through +# Plugins gated by an env var. +# - ``"opt_in"`` plugins refuse to load WITHOUT the env var (e.g. Springer +# needs an API key). +# - ``"opt_out"`` plugins refuse to load WITH the env var set (e.g. IEEE +# and Scholar are default-on; their respective DISABLE env vars flip +# them off). +# list_sources reports enablement without round-tripping through # load_fetcher (which would raise on disabled plugins). -_PLUGIN_ENV_REQUIREMENTS: dict[str, tuple[str, ...]] = { - "ieee": ("AUTOPAPERTOPPT_IEEE_API_KEY", "AUTOPAPERTOPPT_ENABLE_IEEE_SCRAPING"), +_PLUGIN_OPT_IN_ENV: dict[str, tuple[str, ...]] = { "springer": ("AUTOPAPERTOPPT_SPRINGER_API_KEY",), - "scholar": ("AUTOPAPERTOPPT_ENABLE_SCHOLAR_SCRAPING",), +} +_PLUGIN_OPT_OUT_ENV: dict[str, tuple[str, ...]] = { + "ieee": ("AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING",), + "scholar": ("AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING",), } @@ -83,12 +89,16 @@ def _register_discovery_tools(server: FastMCP) -> None: def list_sources() -> dict[str, Any]: """Report every available source plugin and whether it is currently enabled. - A plugin is *enabled* when no env var is required, or when one of - its required env vars is set (the scholar plugin needs - ``AUTOPAPERTOPPT_ENABLE_SCHOLAR_SCRAPING=1``; the springer plugin - needs ``AUTOPAPERTOPPT_SPRINGER_API_KEY``; the ieee plugin needs - either ``AUTOPAPERTOPPT_IEEE_API_KEY`` or - ``AUTOPAPERTOPPT_ENABLE_IEEE_SCRAPING=1``). + Plugin gating today: + + - **ieee** — default-ON. Set ``AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING=1`` + to opt out, or ``AUTOPAPERTOPPT_IEEE_API_KEY`` for the official + Xplore API (better metadata + pdf_url for subscribers). + - **scholar** — default-ON. Set ``AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING=1`` + to opt out (Google's ToS forbids automated access; default-on + for coverage, accept the risk). + - **springer** — opt-IN via ``AUTOPAPERTOPPT_SPRINGER_API_KEY``. + Free key from https://dev.springernature.com/. Agents should call this once before ``search`` so they pass only enabled sources — disabled plugins are silently skipped by the @@ -96,15 +106,19 @@ def list_sources() -> dict[str, Any]: """ entries: list[dict[str, Any]] = [] for name in ALL_SOURCES: - env_vars = _PLUGIN_ENV_REQUIREMENTS.get(name, ()) - enabled = (not env_vars) or any( - _env_var_truthy(var) for var in env_vars + opt_in_vars = _PLUGIN_OPT_IN_ENV.get(name, ()) + opt_out_vars = _PLUGIN_OPT_OUT_ENV.get(name, ()) + opted_in = (not opt_in_vars) or any( + _env_var_set(var) for var in opt_in_vars ) + opted_out = any(_env_var_truthy(var) for var in opt_out_vars) + enabled = opted_in and not opted_out entries.append( { "name": name, "in_default_mix": name in DEFAULT_SOURCES, - "needs_env_var": list(env_vars), + "opt_in_env_var": list(opt_in_vars), + "opt_out_env_var": list(opt_out_vars), "enabled": enabled, } ) @@ -114,13 +128,17 @@ def list_sources() -> dict[str, Any]: } +def _env_var_set(name: str) -> bool: + """True when the env var has any non-empty value.""" + return bool((os.environ.get(name) or "").strip()) + + def _env_var_truthy(name: str) -> bool: - value = (os.environ.get(name) or "").strip() - if not value: - return False - if name.endswith("ENABLE_IEEE_SCRAPING") or name.endswith("ENABLE_SCHOLAR_SCRAPING"): - return value == "1" - return True + """True when the env var is set to exactly ``"1"`` — the convention + for DISABLE flags. Loose values like ``"true"`` are intentionally + NOT honoured so a user has to be deliberate about flipping a default + off.""" + return (os.environ.get(name) or "").strip() == "1" def _register_pdf_tool(server: FastMCP) -> None: diff --git a/docs/architecture.md b/docs/architecture.md index 3afb42e..28425f5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -126,6 +126,14 @@ came back. └──────────┘ │ ▼ + (optional) top-tier filter + │ + ▼ + ┌────────────────┐ + │ oa_resolver │ Unpaywall + arXiv title fallback — + └────────────────┘ fills pdf_url for paywalled-source papers + │ + ▼ (optional) enrich PDF → PaperSummary │ ▼ @@ -137,6 +145,37 @@ came back. └───────────────┘ ``` +### OA PDF resolution + +`autopapertoppt.core.oa_resolver` runs after dedup + rank + top-tier +filter. For every paper still missing `pdf_url`, five strategies fire +in order, returning the first hit: + +1. **arXiv-ID direct** — if the paper carries `arxiv_id` (set by the + openalex / pubmed / crossref / semantic_scholar parsers when the + upstream identified an arXiv preprint), derive + `https://arxiv.org/pdf/{arxiv_id}.pdf` directly. Zero network + round-trip; highest precision; fastest. +2. **Unpaywall** (https://api.unpaywall.org/v2/{doi}) — free, no API + key; needs `AUTOPAPERTOPPT_CONTACT_EMAIL` for politeness. ~50M + papers indexed. +3. **Semantic Scholar OA index** — S2's `openAccessPdf` field is + partially disjoint from Unpaywall; when one misses, the other + often hits. Free, no API key required (rate-limited). +4. **CORE.ac.uk** — aggregator of 200M+ OA repository items + (institutional repos, regional preprint servers, OA journals). + Needs `AUTOPAPERTOPPT_CORE_API_KEY` (free); skipped silently when + unset. +5. **arXiv title search** — for papers without a DOI / arxiv_id, search + arXiv by the paper's title. Exact-match on the normalised title. + +Every lookup is best-effort and never raises; a paper that resists +all five passes through with `pdf_url=None` and the downstream +paywall gate / per-paper renderer falls back to the lightweight tier. + +Disabled per-run via the CLI's `--no-oa-resolve` flag or +`run_search(query, resolve_oa=False)` from Python. + ### Dedup `autopapertoppt.core.dedup` is a three-pass merge: diff --git a/docs/cli.md b/docs/cli.md index fcc2775..6ea7415 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -46,7 +46,8 @@ autopapertoppt (--query KEYWORDS | --paper IDENTIFIER) | `--enrich` | auto-on when `ANTHROPIC_API_KEY` is set | Fetch each paper's PDF and have the Anthropic API write a structured summary; the deck switches to thesis-style layout. Requires `ANTHROPIC_API_KEY` and the `[intelligence]` extra. **Not needed when running over MCP** — an LLM agent can call `fetch_pdf_text` + `export` directly with a hand-crafted summary. | | `--lightweight` | off | Force the abstract-only deck even when `ANTHROPIC_API_KEY` is set. Useful for unattended runs where you do not want to spend tokens. | | `--llm-model` | `claude-opus-4-7` | Override the default model used when `--enrich` is on. Also reads `AUTOPAPERTOPPT_LLM_MODEL`. | -| `--all-venues` | off | Disable the top-tier whitelist. By default the search keeps only flagship CS conferences / journals + Nature / Science / PNAS / CACM / LNCS. arXiv passes through unconditionally. | +| `--top-tier-only` | off | Restrict results to the curated top-tier CS venue whitelist (S&P / CCS / NDSS / USENIX Security / NeurIPS / ICML / ICSE / SIGMOD / SIGCOMM / CHI / etc.) + arXiv pass-through. **Off by default** so IEEE / ACM workshop papers (which dominate "LLM × security" / "LLM × X" topics) survive. | +| `--no-oa-resolve` | off | Skip the open-access PDF resolver step that runs after dedup. By default the pipeline looks up every paper without `pdf_url` in Unpaywall (needs `AUTOPAPERTOPPT_CONTACT_EMAIL`) and falls back to an arXiv title search — typical lift of 40-70% for IEEE / ACM / Springer / Elsevier paywalled papers. Use this flag if you want raw source output without OA enrichment, or to skip the extra HTTP round-trips on a tight latency budget. | | `--paywall-threshold` | `0.30` | Fraction of paywalled results above which the search-mode pipeline asks the user before generating per-paper PPTs. | | `--yes` | off | Auto-accept the paywall prompt. | | `--max-slides` | `25` | Per-paper slide cap. Pass `0` for unlimited. | diff --git a/docs/configuration.md b/docs/configuration.md index 9a88aeb..1d0f7c7 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -24,14 +24,17 @@ each value into `os.environ` before any fetcher initialises. | Variable | Default | Effect | |---|---|---| -| `AUTOPAPERTOPPT_S2_API_KEY` | unset | Higher rate limit on the Semantic Scholar plugin (1/s anonymous → 10/s with key). | +| `AUTOPAPERTOPPT_S2_API_KEY` | unset | Higher rate limit on the Semantic Scholar plugin (1/s anonymous → 10/s with key). **Also used by the OA resolver's S2 `openAccessPdf` lookup step** — without the key the resolver's S2 calls hit the anonymous tier and rate-limit fast. Free key at <https://www.semanticscholar.org/product/api>. | | `AUTOPAPERTOPPT_NCBI_API_KEY` | unset | Raises PubMed's anonymous limit from 3 req/s to 10 req/s. | | `AUTOPAPERTOPPT_IEEE_API_KEY` | unset | Switches the IEEE plugin from the scrape fallback to the official Xplore API (`ieeexploreapi.ieee.org`). Surfaces `pdf_url` for papers in your subscription scope. Apply at <https://developer.ieee.org/>. | -| `AUTOPAPERTOPPT_ENABLE_IEEE_SCRAPING` | unset | Must be `=1` to enable the IEEE scrape fallback. Not needed when `AUTOPAPERTOPPT_IEEE_API_KEY` is set. IEEE Xplore terms of use are grey on automated traffic — opt in deliberately. | +| `AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING` | unset | **IEEE plugin is now default-ON.** Set `=1` to opt out of the scrape fallback. IEEE Xplore ToS are grey on automated traffic — set this if you don't want the scrape path running. | | `AUTOPAPERTOPPT_SPRINGER_API_KEY` | unset | Free key from <https://dev.springernature.com/>. **Required** for the Springer plugin — it raises `ConfigError` at construction without a key, which the pipeline silently skips. | | `AUTOPAPERTOPPT_CROSSREF_PLUS_TOKEN` | unset | Crossref Plus subscriber token. Attached to requests as `Crossref-Plus-API-Token: Bearer <token>`. Raises rate limits and improves cache freshness on the `acm` and `crossref` plugins. | -| `AUTOPAPERTOPPT_ENABLE_SCHOLAR_SCRAPING` | unset | Must be `=1` to enable the Google Scholar plugin. Scholar's terms of use forbid scraping — off by default. | -| `AUTOPAPERTOPPT_CONTACT_EMAIL` | unset | Sent to Crossref / OpenAlex as the `mailto=` parameter (entry into their polite pool) and to NCBI as `tool` / `email` headers. Set this for any non-trivial workload. | +| `AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING` | unset | **Scholar plugin is now default-ON.** Set `=1` to opt out. Google's ToS forbids automated access; default-on for coverage, opt-out if you'd rather not take the captcha / IP-block risk. | +| `AUTOPAPERTOPPT_DISABLE_WEBRUNNER` | unset | **Scholar + IEEE plugins + the PDF downloader for paywalled publisher CDNs all default to driving a real visible Chrome through WebRunner** (`je_web_runner` is a default dependency) — publisher bot-detection is far less aggressive on real browsers. PDF download host list includes `ieeexplore.ieee.org`, `dl.acm.org`, `link.springer.com`, `sciencedirect.com`, `onlinelibrary.wiley.com`, `tandfonline.com`, `academic.oup.com`, `nature.com`, `science.org`, plus a few engineering-society CDNs. Set `=1` to force the httpx paths instead (useful for CI / Docker without a Chrome binary). | +| `AUTOPAPERTOPPT_CHROME_PROFILE_DIR` | unset | When set, passes `--user-data-dir=<path>` to Chrome so cookies / login state survive across CLI invocations. Used by **both** Scholar (one-time Google sign-in suppresses Scholar captchas) and IEEE (institutional auth cookies surface paywalled metadata). | +| `AUTOPAPERTOPPT_CORE_API_KEY` | unset | Free key from <https://core.ac.uk/services/api>. Enables the OA resolver's CORE.ac.uk lookup step (200M+ institutional / regional OA repository items). Skipped silently when unset (the other OA strategies — Unpaywall, Semantic Scholar, arXiv — still run). | +| `AUTOPAPERTOPPT_CONTACT_EMAIL` | unset | Sent to Crossref / OpenAlex as the `mailto=` parameter (entry into their polite pool), to NCBI as `tool` / `email` headers, **and to Unpaywall as `email=`** for the post-dedup OA PDF resolver. Highly recommended — without it the resolver skips Unpaywall lookups entirely, which is the single biggest PDF coverage win for IEEE / ACM / Springer / Elsevier paywalled papers (typical lift 40-70%). | ### PDF download @@ -162,6 +165,50 @@ Override via: Clear the cache by deleting the directory; AutoPaperToPPT re-creates it on demand. +## Suppressing Scholar captchas with a persistent Chrome profile + +Google flags an IP after a few automated Scholar requests even with +WebRunner's real-browser path. The reliable workaround is to seed a +persistent Chrome profile with a real Google sign-in once; subsequent +headless runs reuse the same session cookies, which Google trusts. + +**One-time setup:** + +```powershell +# 1. Pick a directory anywhere on disk +$env:AUTOPAPERTOPPT_CHROME_PROFILE_DIR = "D:\autopapertoppt-scholar-profile" + +# 2. Open Chrome visibly and trigger one Scholar request +$env:AUTOPAPERTOPPT_CHROME_HEADLESS = "0" +autopapertoppt --query "any keywords" --source scholar --max 1 --out .\tmp\ + +# Chrome opens. Sign into your Google account, accept any consent +# banners, complete any captcha. The window holds open for 60s. +``` + +**Every run after that:** + +```powershell +$env:AUTOPAPERTOPPT_CHROME_PROFILE_DIR = "D:\autopapertoppt-scholar-profile" +Remove-Item Env:\AUTOPAPERTOPPT_CHROME_HEADLESS # back to headless +autopapertoppt --query "..." --out .\exports\ +``` + +Chrome boots headless but loads the same profile dir, sends your +authenticated Google session cookie, and Scholar serves real results +instead of a captcha page. + +**Caveats:** + +- Only one Chrome process can hold the profile dir at a time. If you + have a regular Chrome open on the same profile path, the + WebRunner instance will fail to start. Use a dedicated path. +- The session cookie is a real authentication credential. Treat the + profile directory like a secret — back it up if you re-image the + machine, restrict file permissions. +- Cookie eventually expires (~1-2 months for Google). Re-do the + interactive sign-in then. + ## Settings the project explicitly does NOT have By design — listing them so a contributor doesn't accidentally diff --git a/docs/de/index.rst b/docs/de/index.rst index 73db0cc..6848c71 100644 --- a/docs/de/index.rst +++ b/docs/de/index.rst @@ -153,7 +153,7 @@ Weiterführende Quellen * CLI-Flags und Umgebungsvariablen: :doc:`/cli` * 11 MCP-Server-Tools: :doc:`/mcp` * PPTX-Edit-Toolkit: :doc:`/pptx_editing` -* Die Datei ``README.de.md`` im Repo-Root enthält die vollständige +* Die Datei ``readmes/README.de.md`` im Repo-Root enthält die vollständige Feature-Liste. * Die tiefe technische Referenz (Plugin-Architektur, Sicherheitsrichtlinien, Definition of Done, SonarQube-Regeln, …) diff --git a/docs/es/index.rst b/docs/es/index.rst index e505ffd..270453e 100644 --- a/docs/es/index.rst +++ b/docs/es/index.rst @@ -152,7 +152,7 @@ Dónde buscar más * Flags CLI y variables de entorno: :doc:`/cli` * 11 herramientas del servidor MCP: :doc:`/mcp` * Kit de edición PPTX: :doc:`/pptx_editing` -* El archivo ``README.es.md`` en la raíz del repo tiene la lista +* El archivo ``readmes/README.es.md`` en la raíz del repo tiene la lista completa de funcionalidades del proyecto. * La referencia técnica profunda (arquitectura de plugins, políticas de seguridad, Definition of Done, reglas SonarQube, …) está diff --git a/docs/fr/index.rst b/docs/fr/index.rst index c7e6140..46fdfbb 100644 --- a/docs/fr/index.rst +++ b/docs/fr/index.rst @@ -153,7 +153,7 @@ Où chercher plus loin * Flags CLI + variables d'environnement : :doc:`/cli` * 11 outils du serveur MCP : :doc:`/mcp` * Boîte à outils d'édition PPTX : :doc:`/pptx_editing` -* Le fichier ``README.fr.md`` à la racine du repo donne la liste +* Le fichier ``readmes/README.fr.md`` à la racine du repo donne la liste complète des fonctionnalités. * La référence technique approfondie (architecture de plugins, politique de sécurité, Definition of Done, règles SonarQube, …) diff --git a/docs/hi/index.rst b/docs/hi/index.rst index fee7238..ce28fbd 100644 --- a/docs/hi/index.rst +++ b/docs/hi/index.rst @@ -147,7 +147,7 @@ CLI फ़्लैग की पूरी तालिका: :doc:`/cli`। * CLI फ़्लैग और पर्यावरण चर: :doc:`/cli` * 11 MCP सर्वर उपकरण: :doc:`/mcp` * PPTX संपादन टूलकिट: :doc:`/pptx_editing` -* repo जड़ में ``README.hi.md`` फ़ाइल में सुविधाओं की पूरी सूची है। +* repo जड़ में ``readmes/README.hi.md`` फ़ाइल में सुविधाओं की पूरी सूची है। * गहन तकनीकी संदर्भ (प्लगइन वास्तुकला, सुरक्षा नीतियाँ, Definition of Done, SonarQube नियम, …) अंग्रेज़ी गाइड में समेकित हैं: :doc:`/en/index`। diff --git a/docs/id/index.rst b/docs/id/index.rst index 0c17b24..a8e01b7 100644 --- a/docs/id/index.rst +++ b/docs/id/index.rst @@ -149,7 +149,7 @@ Bacaan lebih lanjut * Flag CLI dan variabel lingkungan: :doc:`/cli` * 11 tool server MCP: :doc:`/mcp` * Toolkit edit PPTX: :doc:`/pptx_editing` -* Berkas ``README.id.md`` di akar repo berisi daftar fitur lengkap. +* Berkas ``readmes/README.id.md`` di akar repo berisi daftar fitur lengkap. * Referensi teknis mendalam (arsitektur plugin, kebijakan keamanan, Definition of Done, aturan SonarQube, …) terkonsolidasi di panduan Inggris: :doc:`/en/index`. diff --git a/docs/it/index.rst b/docs/it/index.rst index e6ba82a..85cfefa 100644 --- a/docs/it/index.rst +++ b/docs/it/index.rst @@ -149,7 +149,7 @@ Dove cercare oltre * Flag CLI e variabili d'ambiente: :doc:`/cli` * 11 strumenti del server MCP: :doc:`/mcp` * Toolkit di editing PPTX: :doc:`/pptx_editing` -* Il file ``README.it.md`` nella radice del repo contiene l'elenco +* Il file ``readmes/README.it.md`` nella radice del repo contiene l'elenco completo delle funzionalità. * Il riferimento tecnico approfondito (architettura dei plugin, policy di sicurezza, Definition of Done, regole SonarQube, …) è diff --git a/docs/ja/index.rst b/docs/ja/index.rst index 1f10d92..cc7d65b 100644 --- a/docs/ja/index.rst +++ b/docs/ja/index.rst @@ -148,7 +148,7 @@ CLI フラグの完全な表は :doc:`/cli` を参照してください。 * CLI フラグの完全な一覧と環境変数: :doc:`/cli` * MCP サーバーの 11 ツール: :doc:`/mcp` * PPTX 編集ツールキット: :doc:`/pptx_editing` -* このリポジトリ言語別の README ファイル(``README.ja.md``\ など)に +* このリポジトリ言語別の README ファイル(``readmes/README.ja.md``\ など)に プロジェクトの全機能リストがあります * より詳しい技術リファレンス(プラグインアーキテクチャ、安全性ポリシー、 Definition of Done、SonarQube ルールなど)は英語版ガイド diff --git a/docs/ko/index.rst b/docs/ko/index.rst index eb02fca..39296c7 100644 --- a/docs/ko/index.rst +++ b/docs/ko/index.rst @@ -146,7 +146,7 @@ CLI 플래그 전체 표: :doc:`/cli`. * CLI 플래그 + 환경 변수: :doc:`/cli` * 11 개 MCP 서버 도구: :doc:`/mcp` * PPTX 편집 툴킷: :doc:`/pptx_editing` -* repo 루트의 ``README.ko.md`` 에 기능 전체 목록이 있습니다. +* repo 루트의 ``readmes/README.ko.md`` 에 기능 전체 목록이 있습니다. * 깊이 있는 기술 참조 (플러그인 아키텍처, 보안 정책, Definition of Done, SonarQube 규칙 등) 는 영어 가이드에 집중되어 있습니다: :doc:`/en/index`. diff --git a/docs/pt/index.rst b/docs/pt/index.rst index f5af401..2e1cebc 100644 --- a/docs/pt/index.rst +++ b/docs/pt/index.rst @@ -149,7 +149,7 @@ Onde procurar mais * Flags CLI e variáveis de ambiente: :doc:`/cli` * 11 ferramentas do servidor MCP: :doc:`/mcp` * Toolkit de edição PPTX: :doc:`/pptx_editing` -* O arquivo ``README.pt.md`` na raiz do repo tem a lista completa de +* O arquivo ``readmes/README.pt.md`` na raiz do repo tem a lista completa de funcionalidades. * A referência técnica profunda (arquitetura de plugins, políticas de segurança, Definition of Done, regras SonarQube, …) está diff --git a/docs/ru/index.rst b/docs/ru/index.rst index 4c61daa..cafcac9 100644 --- a/docs/ru/index.rst +++ b/docs/ru/index.rst @@ -151,7 +151,7 @@ DOI, столбец 8 = URL. Прогоните аудит после regen-ск * Флаги CLI и переменные окружения: :doc:`/cli` * 11 инструментов MCP-сервера: :doc:`/mcp` * Инструменты редактирования PPTX: :doc:`/pptx_editing` -* В файле ``README.ru.md`` в корне репозитория есть полный список +* В файле ``readmes/README.ru.md`` в корне репозитория есть полный список возможностей проекта. * Глубокий технический справочник (архитектура плагинов, политики безопасности, Definition of Done, правила SonarQube, …) diff --git a/docs/vi/index.rst b/docs/vi/index.rst index eace6fc..d81b665 100644 --- a/docs/vi/index.rst +++ b/docs/vi/index.rst @@ -147,7 +147,7 @@ Tìm hiểu thêm * Cờ CLI và biến môi trường: :doc:`/cli` * 11 công cụ máy chủ MCP: :doc:`/mcp` * Toolkit chỉnh sửa PPTX: :doc:`/pptx_editing` -* Tệp ``README.vi.md`` ở gốc repo có danh sách đầy đủ tính năng. +* Tệp ``readmes/README.vi.md`` ở gốc repo có danh sách đầy đủ tính năng. * Tham chiếu kỹ thuật sâu (kiến trúc plugin, chính sách bảo mật, Definition of Done, luật SonarQube, …) được tập trung trong hướng dẫn tiếng Anh: :doc:`/en/index`. diff --git a/pyproject.toml b/pyproject.toml index 06c3316..ccad595 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -44,6 +44,11 @@ dependencies = [ "beautifulsoup4>=4.12", "lxml>=5.2", "markdown-it-py>=3.0", + # Real-browser backend for the Google Scholar plugin. Default-on so + # the out-of-box `pip install autopapertoppt` gets the captcha-resilient + # Scholar path. Users who don't have Chrome on PATH automatically + # fall through to the httpx scrape path with no breakage. + "je_web_runner>=0.0.60", ] [project.optional-dependencies] diff --git a/README.de.md b/readmes/README.de.md similarity index 97% rename from README.de.md rename to readmes/README.de.md index 3f41d03..2cbbcca 100644 --- a/README.de.md +++ b/readmes/README.de.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **Sprachen**: [English](README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · **Deutsch** · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) +> **Sprachen**: [English](../README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · **Deutsch** · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) > **Dokumentation**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) Stichwortgesteuerter Paper-Such-Assistent, der Ergebnisse von arXiv, Semantic Scholar, OpenAlex, PubMed, ACM (via Crossref), IEEE Xplore, DBLP, generischem Crossref, OpenAIRE, Springer Nature und Google Scholar abruft, in ein einheitliches Datensatzformat normalisiert und die deduplizierte Ergebnismenge als **Paper-Review PowerPoint im Thesis-Stil**, **Excel-Arbeitsmappe** und **BibTeX-Datei** exportiert — alles aus einem CLI-Aufruf oder einem MCP-Tool-Aufruf. Kann optional jedes Paper anreichern, indem es das PDF liest und eine strukturierte Zusammenfassung erstellt, entweder im Kontext (LLM-as-agent-Pfad) oder über die Anthropic-API (Python-Pipeline-Pfad). diff --git a/README.es.md b/readmes/README.es.md similarity index 97% rename from README.es.md rename to readmes/README.es.md index 2908df6..ca6f4dd 100644 --- a/README.es.md +++ b/readmes/README.es.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **Idiomas**: [English](README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · **Español** · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) +> **Idiomas**: [English](../README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · **Español** · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) > **Documentación**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) Asistente de búsqueda de artículos guiado por palabras clave que recupera resultados de arXiv, Semantic Scholar, OpenAlex, PubMed, ACM (vía Crossref), IEEE Xplore, DBLP, Crossref genérico, OpenAIRE, Springer Nature y Google Scholar; los normaliza a un único formato de registro y exporta el conjunto deduplicado como una **presentación PowerPoint estilo tesis**, un **libro Excel** y un **archivo BibTeX** — todo desde una llamada CLI o una llamada de herramienta MCP. Opcionalmente enriquece cada artículo leyendo su PDF y produciendo un resumen estructurado, ya sea en contexto (flujo LLM-as-agent) o mediante la API de Anthropic (flujo Python pipeline). diff --git a/README.fr.md b/readmes/README.fr.md similarity index 97% rename from README.fr.md rename to readmes/README.fr.md index 21f0abb..2575be5 100644 --- a/README.fr.md +++ b/readmes/README.fr.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **Langues** : [English](README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · **Français** · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) +> **Langues** : [English](../README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · **Français** · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) > **Documentation** : [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) Assistant de recherche d'articles piloté par mots-clés. Il interroge arXiv, Semantic Scholar, OpenAlex, PubMed, ACM (via Crossref), IEEE Xplore, DBLP, Crossref générique, OpenAIRE, Springer Nature et Google Scholar ; normalise les résultats en un format de fiche unique ; et exporte l'ensemble dédupliqué en **présentation PowerPoint de style thèse**, **classeur Excel** et **fichier BibTeX** — le tout par un seul appel CLI ou un seul appel MCP. Peut également enrichir chaque article en lisant son PDF pour produire un résumé structuré, soit en contexte (flux LLM-as-agent), soit via l'API Anthropic (flux Python pipeline). diff --git a/README.hi.md b/readmes/README.hi.md similarity index 98% rename from README.hi.md rename to readmes/README.hi.md index 2a270a3..0ab6e5a 100644 --- a/README.hi.md +++ b/readmes/README.hi.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **भाषाएँ**: [English](README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · **हिन्दी** · [Bahasa Indonesia](README.id.md) +> **भाषाएँ**: [English](../README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · **हिन्दी** · [Bahasa Indonesia](README.id.md) > **दस्तावेज़ीकरण**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) कीवर्ड-संचालित शोध-पत्र खोज सहायक। arXiv, Semantic Scholar, OpenAlex, PubMed, ACM (Crossref के माध्यम से), IEEE Xplore, DBLP, सामान्य Crossref, OpenAIRE, Springer Nature और Google Scholar से परिणाम लाता है; उन्हें एकल रिकॉर्ड प्रारूप में सामान्यीकृत करता है; और डुप्लीकेट-मुक्त समूह को **थीसिस-शैली PowerPoint स्लाइड**, **Excel वर्कबुक** और **BibTeX फ़ाइल** के रूप में निर्यात करता है — एक CLI कॉल या एक MCP टूल कॉल से सब कुछ। वैकल्पिक रूप से प्रत्येक शोध-पत्र को उसकी PDF पढ़कर समृद्ध कर सकता है, या तो संदर्भ में (LLM-as-agent पथ) या Anthropic API के माध्यम से (Python pipeline पथ)। diff --git a/README.id.md b/readmes/README.id.md similarity index 97% rename from README.id.md rename to readmes/README.id.md index 967e004..2e4b0b9 100644 --- a/README.id.md +++ b/readmes/README.id.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **Bahasa**: [English](README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · **Bahasa Indonesia** +> **Bahasa**: [English](../README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · **Bahasa Indonesia** > **Dokumentasi**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) Asisten pencarian makalah berbasis kata kunci. Mengambil hasil dari arXiv, Semantic Scholar, OpenAlex, PubMed, ACM (via Crossref), IEEE Xplore, DBLP, Crossref umum, OpenAIRE, Springer Nature, dan Google Scholar; menormalkannya ke satu format catatan; dan mengekspor kumpulan yang telah dideduplikasi sebagai **slide PowerPoint gaya tesis**, **buku kerja Excel**, dan **berkas BibTeX** — semua dari satu panggilan CLI atau satu panggilan tool MCP. Opsional, dapat memperkaya setiap makalah dengan membaca PDF-nya dan menghasilkan ringkasan terstruktur, baik dalam konteks (alur LLM-as-agent) atau via API Anthropic (alur Python pipeline). diff --git a/README.it.md b/readmes/README.it.md similarity index 97% rename from README.it.md rename to readmes/README.it.md index d8724a0..fa1aa27 100644 --- a/README.it.md +++ b/readmes/README.it.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **Lingue**: [English](README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · **Italiano** · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) +> **Lingue**: [English](../README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · **Italiano** · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) > **Documentazione**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) Assistente di ricerca di articoli guidato da parole chiave. Recupera risultati da arXiv, Semantic Scholar, OpenAlex, PubMed, ACM (via Crossref), IEEE Xplore, DBLP, Crossref generico, OpenAIRE, Springer Nature e Google Scholar; li normalizza in un unico formato di record; ed esporta l'insieme deduplicato come **presentazione PowerPoint stile tesi**, **cartella di lavoro Excel** e **file BibTeX** — tutto da una chiamata CLI o un'invocazione MCP. Può arricchire ciascun articolo leggendone il PDF e producendo un riassunto strutturato, sia in-contesto (flusso LLM-as-agent) sia tramite API Anthropic (flusso Python pipeline). diff --git a/README.ja.md b/readmes/README.ja.md similarity index 97% rename from README.ja.md rename to readmes/README.ja.md index 07ad06e..b29f985 100644 --- a/README.ja.md +++ b/readmes/README.ja.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **言語**: [English](README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · **日本語** · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) +> **言語**: [English](../README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · **日本語** · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) > **ドキュメント**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) キーワード駆動の論文検索アシスタント。arXiv、Semantic Scholar、OpenAlex、PubMed、ACM(Crossref 経由)、IEEE Xplore、DBLP、汎用 Crossref、OpenAIRE、Springer Nature、Google Scholar から論文を取得し、統一されたレコード形式に正規化、重複排除後の結果集合を **論文発表用 PowerPoint スライド**、**Excel ワークブック**、**BibTeX ファイル** として出力します — CLI 1 回または MCP ツール呼び出し 1 回で完結。各論文の PDF を読んで構造化サマリを生成することも可能で、LLM-as-agent パスまたは Anthropic API パスから選べます。 diff --git a/README.ko.md b/readmes/README.ko.md similarity index 97% rename from README.ko.md rename to readmes/README.ko.md index aaf82bc..0e7e513 100644 --- a/README.ko.md +++ b/readmes/README.ko.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **언어**: [English](README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · **한국어** · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) +> **언어**: [English](../README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · **한국어** · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) > **문서**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) 키워드 기반 논문 검색 어시스턴트. arXiv, Semantic Scholar, OpenAlex, PubMed, ACM (Crossref 경유), IEEE Xplore, DBLP, 일반 Crossref, OpenAIRE, Springer Nature, Google Scholar 에서 결과를 가져와 단일 레코드 형식으로 정규화하고, 중복 제거된 결과를 **논문 발표용 PowerPoint 슬라이드**, **Excel 워크북**, **BibTeX 파일** 로 내보냅니다 — CLI 호출 한 번 또는 MCP 도구 호출 한 번으로 끝납니다. 각 논문의 PDF 를 읽고 구조화된 요약을 생성할 수도 있으며 (LLM-as-agent 경로), 또는 Anthropic API 경유 (Python 파이프라인 경로) 가능합니다. diff --git a/README.pt.md b/readmes/README.pt.md similarity index 97% rename from README.pt.md rename to readmes/README.pt.md index 5a4d840..a7e2d97 100644 --- a/README.pt.md +++ b/readmes/README.pt.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **Idiomas**: [English](README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · **Português** · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) +> **Idiomas**: [English](../README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · **Português** · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) > **Documentação**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) Assistente de busca de artigos guiado por palavras-chave que recupera resultados do arXiv, Semantic Scholar, OpenAlex, PubMed, ACM (via Crossref), IEEE Xplore, DBLP, Crossref genérico, OpenAIRE, Springer Nature e Google Scholar; normaliza-os para um único formato de registro; e exporta o conjunto deduplicado como **apresentação PowerPoint estilo tese**, **planilha Excel** e **arquivo BibTeX** — tudo por uma única chamada CLI ou uma chamada de ferramenta MCP. Pode opcionalmente enriquecer cada artigo lendo seu PDF e produzindo um resumo estruturado, no próprio contexto (fluxo LLM-as-agent) ou via API Anthropic (fluxo Python pipeline). diff --git a/README.ru.md b/readmes/README.ru.md similarity index 98% rename from README.ru.md rename to readmes/README.ru.md index 8838a5b..a482a9f 100644 --- a/README.ru.md +++ b/readmes/README.ru.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **Языки**: [English](README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · **Русский** · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) +> **Языки**: [English](../README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · **Русский** · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) > **Документация**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) Поисковый ассистент статей, управляемый ключевыми словами. Получает результаты из arXiv, Semantic Scholar, OpenAlex, PubMed, ACM (через Crossref), IEEE Xplore, DBLP, общего Crossref, OpenAIRE, Springer Nature и Google Scholar; нормализует в единый формат записи; и экспортирует дедуплицированный набор как **слайды PowerPoint в стиле дипломной презентации**, **книгу Excel** и **файл BibTeX** — всё за один CLI-вызов или один вызов MCP-инструмента. Опционально обогащает каждую статью, читая её PDF и порождая структурированную сводку — либо в контексте (поток LLM-as-agent), либо через API Anthropic (Python pipeline). diff --git a/README.vi.md b/readmes/README.vi.md similarity index 97% rename from README.vi.md rename to readmes/README.vi.md index ff6b52a..2f367eb 100644 --- a/README.vi.md +++ b/readmes/README.vi.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **Ngôn ngữ**: [English](README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · **Tiếng Việt** · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) +> **Ngôn ngữ**: [English](../README.md) · [繁體中文](README.zh-TW.md) · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · **Tiếng Việt** · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) > **Tài liệu**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) Trợ lý tìm kiếm bài báo theo từ khóa. Lấy kết quả từ arXiv, Semantic Scholar, OpenAlex, PubMed, ACM (qua Crossref), IEEE Xplore, DBLP, Crossref tổng quát, OpenAIRE, Springer Nature và Google Scholar; chuẩn hóa về một định dạng bản ghi duy nhất; và xuất tập đã khử trùng lặp thành **slide PowerPoint phong cách luận văn**, **sổ Excel** và **tệp BibTeX** — tất cả từ một lệnh CLI hoặc một lời gọi công cụ MCP. Có thể làm giàu mỗi bài báo bằng cách đọc PDF và tạo bản tóm tắt có cấu trúc, ngay trong ngữ cảnh (luồng LLM-as-agent) hoặc qua API Anthropic (luồng Python pipeline). diff --git a/README.zh-CN.md b/readmes/README.zh-CN.md similarity index 97% rename from README.zh-CN.md rename to readmes/README.zh-CN.md index d359fa2..4f172c2 100644 --- a/README.zh-CN.md +++ b/readmes/README.zh-CN.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **语言**: [English](README.md) · [繁體中文](README.zh-TW.md) · **简体中文** · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) +> **语言**: [English](../README.md) · [繁體中文](README.zh-TW.md) · **简体中文** · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) > **文档**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) 以关键词驱动的论文搜索助手。从 arXiv、Semantic Scholar、OpenAlex、PubMed、ACM(走 Crossref)、IEEE Xplore、DBLP、通用 Crossref、OpenAIRE、Springer Nature、Google Scholar 抓论文,规范化为统一的 record,并把去重后的结果集导出为 **论文答辩级的 PowerPoint 幻灯片**、**Excel 工作簿**、**BibTeX 文件** —— 一次 CLI 调用或一次 MCP 工具调用即可完成全部。可选让 AI 读 PDF 正文后产出每篇论文的结构化摘要(LLM-as-agent 路径)或通过 Anthropic API 自动产(Python pipeline 路径)。 diff --git a/README.zh-TW.md b/readmes/README.zh-TW.md similarity index 97% rename from README.zh-TW.md rename to readmes/README.zh-TW.md index 50796e8..dadfe74 100644 --- a/README.zh-TW.md +++ b/readmes/README.zh-TW.md @@ -7,7 +7,7 @@ [![License: MIT](https://img.shields.io/github/license/Integration-Automation/AutoPaperToPPT.svg)](https://github.com/Integration-Automation/AutoPaperToPPT/blob/main/LICENSE) [![Docs](https://readthedocs.org/projects/autopapertoppt/badge/?version=latest)](https://autopapertoppt.readthedocs.io/en/latest/) -> **語言**: [English](README.md) · **繁體中文** · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) +> **語言**: [English](../README.md) · **繁體中文** · [简体中文](README.zh-CN.md) · [日本語](README.ja.md) · [Español](README.es.md) · [Français](README.fr.md) · [Deutsch](README.de.md) · [한국어](README.ko.md) · [Português](README.pt.md) · [Русский](README.ru.md) · [Italiano](README.it.md) · [Tiếng Việt](README.vi.md) · [हिन्दी](README.hi.md) · [Bahasa Indonesia](README.id.md) > **文件**: [autopapertoppt.readthedocs.io](https://autopapertoppt.readthedocs.io/en/latest/) 以關鍵字驅動的論文搜尋助手。從 arXiv、Semantic Scholar、OpenAlex、PubMed、ACM(走 Crossref)、IEEE Xplore、DBLP、通用 Crossref、OpenAIRE、Springer Nature、Google Scholar 抓論文,正規化成統一的 record,並把去重後的結果集匯出為 **論文口試級的 PowerPoint 投影片**、**Excel 工作簿**、**BibTeX 檔** —— 一次 CLI 呼叫或一次 MCP 工具呼叫即可完成全部。另可選擇讓 AI 讀 PDF 本文後產出每篇論文的結構化摘要(LLM-as-agent 路徑)或透過 Anthropic API 自動產(Python pipeline 路徑)。 diff --git a/scripts/_dump_pdf_text.py b/scripts/_dump_pdf_text.py new file mode 100644 index 0000000..65db23d --- /dev/null +++ b/scripts/_dump_pdf_text.py @@ -0,0 +1,20 @@ +"""Throwaway: dump a PDF's body text so the LLM can read it via the Read tool. + +Usage: + .venv\\Scripts\\python.exe -m scripts._dump_pdf_text <pdf_path> + +Writes ``<pdf_path>.txt`` next to the PDF and prints the first 2 KB. +""" +import sys +from pathlib import Path + +from autopapertoppt.intelligence.pdf import _extract_text + +pdf = Path(sys.argv[1]) +body = pdf.read_bytes() +text, pages = _extract_text(body, str(pdf)) +out = pdf.with_suffix(".txt") +out.write_text(text, encoding="utf-8") +print(f"pdf={pdf.name} pages={pages} chars={len(text)} -> {out}") +print("--- HEAD ---") +print(text[:2000]) diff --git a/scripts/_inspect_xlsx.py b/scripts/_inspect_xlsx.py new file mode 100644 index 0000000..6b891d1 --- /dev/null +++ b/scripts/_inspect_xlsx.py @@ -0,0 +1,18 @@ +"""Throwaway: print every row of a Papers sheet for the LLM to inspect.""" +import sys + +from openpyxl import load_workbook + +wb = load_workbook(sys.argv[1], read_only=True, data_only=True) +ws = wb["Papers"] +rows = list(ws.iter_rows(values_only=True)) +hdr = rows[0] +title_i = hdr.index("Title") +via_i = hdr.index("Indexed via") +doi_i = hdr.index("DOI") +url_i = hdr.index("URL") +for i, r in enumerate(rows[1:], 1): + title = (r[title_i] or "")[:60] + print(f"[{i}] via={r[via_i]:8} | {title}") + print(f" URL={r[url_i]}") + print(f" DOI={r[doi_i]}") diff --git a/scripts/_overflow_check.py b/scripts/_overflow_check.py new file mode 100644 index 0000000..d3722c1 --- /dev/null +++ b/scripts/_overflow_check.py @@ -0,0 +1,122 @@ +"""Headless overflow inspection for one or more .pptx decks. + +Walks every shape on every slide, estimates its rendered (wrapped) text +height with a per-font-size char-per-line heuristic, and flags shapes +that (a) overflow their declared box height or (b) extend past the +7.05" footer guard (where page numbers and slide footer live). + +Usage: + .venv\\Scripts\\python.exe -m scripts._overflow_check <pptx_path> [<pptx_path> ...] + +Exit code 0 = all decks PASS, 1 = any violation found. +""" +from __future__ import annotations + +import sys +from pathlib import Path + +from pptx import Presentation +from pptx.enum.text import MSO_AUTO_SIZE +from pptx.util import Emu + +# 7.05" in EMU — the body guard line. Body content (title / meta / +# subhead / body / kpi / paper_subtitle / rq_box) must not extend +# past this. The page_number and footer shapes are deliberately in +# the band beyond 7.05" — those are the footer territory itself. +FOOTER_GUARD_EMU = int(7.05 * 914400) +_FOOTER_BAND_SHAPES = frozenset({"page_number", "footer"}) + +# Approx chars per inch at given font sizes. Rough but matches the +# project's existing decks' wrap behaviour for default body text. +_CHARS_PER_INCH = { + 9: 14.0, 10: 12.5, 11: 11.5, 12: 10.5, 14: 9.0, + 16: 8.0, 18: 7.0, 20: 6.5, 24: 5.5, 28: 4.8, 30: 4.5, 36: 3.8, +} +_LINE_HEIGHT_FACTOR = 1.22 # line-height multiplier above raw font size + + +def _font_size_pt(run) -> int: + sz = run.font.size + if sz is None: + return 12 + return max(8, sz.pt) + + +def _estimate_wrapped_height_emu(shape) -> int: + tf = shape.text_frame + width_in = (shape.width or Emu(0)) / 914400 or 5.0 + total_lines = 0.0 + weighted_lh_pt = 0.0 + for para in tf.paragraphs: + text = "".join(r.text or "" for r in para.runs) or para.text or "" + if not text: + total_lines += 1 + weighted_lh_pt += 12.0 * _LINE_HEIGHT_FACTOR + continue + runs = list(para.runs) + sz = _font_size_pt(runs[0]) if runs else 12 + cpi = _CHARS_PER_INCH.get(int(sz), 10.5) + chars_per_line = max(1, int(cpi * width_in)) + text_lines = max(1, -(-len(text) // chars_per_line)) # ceil div + total_lines += text_lines + weighted_lh_pt += sz * _LINE_HEIGHT_FACTOR * text_lines + avg_line_height_pt = weighted_lh_pt / max(1.0, total_lines) + return int(avg_line_height_pt / 72.0 * 914400 * total_lines) + + +def _inspect(pptx_path: Path) -> list[tuple[int, str, str, int, int]]: + """Returns a list of (slide_idx, shape_name, kind, rendered, limit).""" + prs = Presentation(pptx_path) + violations: list[tuple[int, str, str, int, int]] = [] + for idx, slide in enumerate(prs.slides, start=1): + for shape in slide.shapes: + if not shape.has_text_frame: + continue + name = shape.name or "?" + top = shape.top or 0 + height = shape.height or 0 + auto = shape.text_frame.auto_size + rendered = _estimate_wrapped_height_emu(shape) + # TEXT_TO_FIT_SHAPE = PowerPoint will auto-shrink text to box; + # SHAPE_TO_FIT_TEXT = PowerPoint will grow the box. Neither + # produces a hard overflow at render time, so the height check + # only applies when auto_size is NONE / disabled. + strict = auto in (None, MSO_AUTO_SIZE.NONE) + bottom = top + (rendered if strict else min(rendered, height)) + if strict and height and rendered > height: + violations.append((idx, name, "overflows box", rendered, height)) + # page_number / footer shapes legitimately sit in the + # footer band — don't flag them. + if name in _FOOTER_BAND_SHAPES: + continue + if bottom > FOOTER_GUARD_EMU: + violations.append((idx, name, "past footer guard", bottom, FOOTER_GUARD_EMU)) + return violations + + +def main() -> int: + if len(sys.argv) < 2: + print("usage: python -m scripts._overflow_check <pptx> [<pptx> ...]") + return 2 + bad = 0 + for arg in sys.argv[1:]: + pptx_path = Path(arg) + prs = Presentation(pptx_path) + violations = _inspect(pptx_path) + slides = len(list(prs.slides)) + shapes = sum(len(list(s.shapes)) for s in prs.slides) + print(f"\noverflow check -- {pptx_path}") + print(f" slides: {slides} shapes: {shapes} violations: {len(violations)}") + for idx, name, kind, rendered, limit in violations: + ren_in = rendered / 914400 + lim_in = limit / 914400 + print(f" slide {idx} shape {name!r}: {kind} -- {ren_in:.2f}\" vs {lim_in:.2f}\"") + verdict = "PASS" if not violations else "FAIL" + print(f" verdict: {verdict}") + if violations: + bad += 1 + return 0 if bad == 0 else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/_pdf_downloaders.py b/scripts/_pdf_downloaders.py new file mode 100644 index 0000000..298fbdf --- /dev/null +++ b/scripts/_pdf_downloaders.py @@ -0,0 +1,555 @@ +"""Per-publisher PDF download helpers (shared by the LLM-driven scripts). + +Each ``download_*`` helper runs against a Chrome driver the caller +already booted, so a batch script can solve a captcha / SSO once at +the start and burn through N papers in the same session instead of +booting Chrome per paper. + +Common contract for each helper: + +* Inputs: ``driver`` (already booted via ``webrunner_browser.make_driver`` + with ``download_dir`` configured), the per-paper identifier, ``out_dir`` + (where the resulting PDF should live). +* Behaviour: clear stale ``*.crdownload`` and any same-named PDF first, + navigate to the landing URL, give the publisher a chance to either + auto-download or expose a PDF link, wait on the download dir, validate + ``%PDF-`` head + ``%%EOF`` tail, rename to the canonical ``<id>.pdf``. +* Return: the saved ``Path`` on success, ``None`` on failure (no PDF + appeared, or the file was not a valid PDF — usually the publisher + served an HTML "Sign in / Get access" gate to an unauthenticated + visitor). + +The helpers do NOT call ``driver.quit()`` — the caller owns the driver +lifecycle so a batch can reuse one Chrome across many papers. +""" + +from __future__ import annotations + +import contextlib +import re +import time +from pathlib import Path +from typing import Any + +from autopapertoppt.fetchers import webrunner_browser + +_DOWNLOAD_POLL_INTERVAL = 1.0 +_DOWNLOAD_MAX_WAIT = 90.0 +_DOC_RENDER_WAIT = 4.0 +_STAMP_RENDER_WAIT = 6.0 + + +def _clear_pending(out_dir: Path) -> None: + """Remove stale .crdownload so a half-finished prior run doesn't trip the wait. + + Deliberately does NOT touch existing .pdf files — earlier papers in a batch + already wrote their final PDFs here under canonical names; wiping them now + would defeat the whole point of batching. + """ + for old in out_dir.glob("*.crdownload"): + with contextlib.suppress(OSError): + old.unlink() + + +def _snapshot_pdfs(out_dir: Path) -> set[Path]: + """Snapshot the existing .pdf set so we can later detect the new arrival.""" + return set(out_dir.glob("*.pdf")) + + +def _wait_for_new_pdf( + out_dir: Path, baseline: set[Path], deadline: float, +) -> Path | None: + """Block until a NEW .pdf lands (not in ``baseline``) and no .crdownload remains.""" + while time.monotonic() < deadline: + pending = list(out_dir.glob("*.crdownload")) + new_pdfs = [p for p in out_dir.glob("*.pdf") if p not in baseline] + if new_pdfs and not pending: + return new_pdfs[0] + time.sleep(_DOWNLOAD_POLL_INTERVAL) + return None + + +def _is_valid_pdf(path: Path) -> bool: + """Magic-header + EOF check — rejects HTML masquerading as PDF.""" + try: + data = path.read_bytes() + except OSError: + return False + if len(data) < 32: + return False + if data[:4] != b"%PDF": + return False + return b"%%EOF" in data[-64:] + + +def _finalise(pdf: Path, canonical_name: str) -> Path | None: + """Validate + rename. Returns the canonical path on success.""" + if not _is_valid_pdf(pdf): + head = pdf.read_bytes()[:8] if pdf.exists() else b"" + size = pdf.stat().st_size if pdf.exists() else 0 + print( + f"[fail] file {pdf.name} ({size} bytes) is not a valid PDF " + f"(head={head!r}). Publisher likely served an HTML gate.", + flush=True, + ) + return None + target = pdf.parent / canonical_name + if pdf != target: + if target.exists(): + target.unlink() + pdf.rename(target) + print( + f"[ok] {target.name} ({target.stat().st_size:,} bytes)", + flush=True, + ) + return target + + +# --------------------------------------------------------------------------- +# IEEE Xplore +# --------------------------------------------------------------------------- + +_IEEE_DOC_URL = "https://ieeexplore.ieee.org/document/{arnumber}" +_IEEE_STAMP_URL = "https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber={arnumber}" +_IEEE_ARNUMBER_RE = re.compile(r"/document/(\d+)") + + +def arnumber_from_url(url: str) -> str | None: + """Pull the IEEE arnumber out of a `/document/<id>` URL.""" + if not url: + return None + m = _IEEE_ARNUMBER_RE.search(url) + return m.group(1) if m else None + + +def download_ieee( + driver: Any, arnumber: str, out_dir: Path, +) -> Path | None: + """Drive Chrome to download an IEEE Xplore PDF by arnumber.""" + target = out_dir / f"{arnumber}.pdf" + if target.exists() and _is_valid_pdf(target): + print(f"[ieee] cached {target.name}", flush=True) + return target + _clear_pending(out_dir) + baseline = _snapshot_pdfs(out_dir) + + doc_url = _IEEE_DOC_URL.format(arnumber=arnumber) + print(f"[ieee] doc {doc_url}", flush=True) + driver.get(doc_url) + time.sleep(_DOC_RENDER_WAIT) + webrunner_browser.wait_for_captcha_solved(driver, max_wait_seconds=300.0) + + stamp_url = _IEEE_STAMP_URL.format(arnumber=arnumber) + print(f"[ieee] stamp {stamp_url}", flush=True) + driver.get(stamp_url) + time.sleep(_STAMP_RENDER_WAIT) + webrunner_browser.wait_for_captcha_solved(driver, max_wait_seconds=300.0) + + deadline = time.monotonic() + _DOWNLOAD_MAX_WAIT + pdf = _wait_for_new_pdf(out_dir, baseline, deadline) + if pdf is None: + # stamp.jsp serves an iframe wrapper; chase the iframe src. + try: + src = driver.execute_script( + "const f=document.querySelector('frame,iframe');" + "return f?f.src:null;" + ) + except Exception: # noqa: BLE001 + src = None + if src and src.startswith("https://"): + print(f"[ieee] iframe retry {src}", flush=True) + driver.get(src) + deadline = time.monotonic() + _DOWNLOAD_MAX_WAIT + pdf = _wait_for_new_pdf(out_dir, baseline, deadline) + + if pdf is None: + print("[ieee] no PDF appeared (paper may be early-access / withdrawn / no subscription access)", flush=True) + return None + return _finalise(pdf, f"{arnumber}.pdf") + + +# --------------------------------------------------------------------------- +# ACM Digital Library +# --------------------------------------------------------------------------- + +_ACM_LANDING_URL = "https://dl.acm.org/doi/{doi}" +_ACM_PDF_URL = "https://dl.acm.org/doi/pdf/{doi}" + + +def _safe_doi_slug(doi: str) -> str: + """Make a DOI safe for use as a filename stem.""" + return doi.replace("/", "_").replace(":", "_") + + +def download_acm(driver: Any, doi: str, out_dir: Path) -> Path | None: + """Drive Chrome to download an ACM-hosted PDF by DOI.""" + canonical = f"{_safe_doi_slug(doi)}.pdf" + target = out_dir / canonical + if target.exists() and _is_valid_pdf(target): + print(f"[acm] cached {target.name}", flush=True) + return target + _clear_pending(out_dir) + baseline = _snapshot_pdfs(out_dir) + + landing = _ACM_LANDING_URL.format(doi=doi) + print(f"[acm] landing {landing}", flush=True) + driver.get(landing) + time.sleep(_DOC_RENDER_WAIT) + webrunner_browser.wait_for_captcha_solved(driver, max_wait_seconds=300.0) + + pdf_url = _ACM_PDF_URL.format(doi=doi) + print(f"[acm] pdf {pdf_url}", flush=True) + driver.get(pdf_url) + time.sleep(_STAMP_RENDER_WAIT) + webrunner_browser.wait_for_captcha_solved(driver, max_wait_seconds=300.0) + + deadline = time.monotonic() + _DOWNLOAD_MAX_WAIT + pdf = _wait_for_new_pdf(out_dir, baseline, deadline) + if pdf is None: + try: + src = driver.execute_script( + "const f=document.querySelector('frame,iframe');" + "return f?f.src:null;" + ) + except Exception: # noqa: BLE001 + src = None + if src and src.startswith("https://"): + print(f"[acm] iframe retry {src}", flush=True) + driver.get(src) + deadline = time.monotonic() + _DOWNLOAD_MAX_WAIT + pdf = _wait_for_new_pdf(out_dir, baseline, deadline) + + if pdf is None: + print("[acm] no PDF appeared (likely paywall — institutional access required)", flush=True) + return None + return _finalise(pdf, canonical) + + +# --------------------------------------------------------------------------- +# SpringerLink +# --------------------------------------------------------------------------- + +_SPRINGER_ARTICLE_URL = "https://link.springer.com/article/{doi}" +_SPRINGER_CHAPTER_URL = "https://link.springer.com/chapter/{doi}" +_SPRINGER_PDF_URL = "https://link.springer.com/content/pdf/{doi}.pdf" + + +def download_springer(driver: Any, doi: str, out_dir: Path) -> Path | None: + """Drive Chrome to download a SpringerLink PDF by DOI. + + Springer hosts both journal articles (`/article/<doi>`) and book + chapters (`/chapter/<doi>`); we try article first and fall through + to chapter on a 404 / "page not found" body. + """ + canonical = f"{_safe_doi_slug(doi)}.pdf" + target = out_dir / canonical + if target.exists() and _is_valid_pdf(target): + print(f"[spr] cached {target.name}", flush=True) + return target + _clear_pending(out_dir) + baseline = _snapshot_pdfs(out_dir) + + article = _SPRINGER_ARTICLE_URL.format(doi=doi) + print(f"[spr] article {article}", flush=True) + driver.get(article) + time.sleep(_DOC_RENDER_WAIT) + webrunner_browser.wait_for_captcha_solved(driver, max_wait_seconds=300.0) + if "Page not found" in (driver.page_source or "")[:8192]: + chapter = _SPRINGER_CHAPTER_URL.format(doi=doi) + print(f"[spr] not an article, retrying chapter {chapter}", flush=True) + driver.get(chapter) + time.sleep(_DOC_RENDER_WAIT) + webrunner_browser.wait_for_captcha_solved(driver, max_wait_seconds=300.0) + + pdf_url = _SPRINGER_PDF_URL.format(doi=doi) + print(f"[spr] pdf {pdf_url}", flush=True) + driver.get(pdf_url) + time.sleep(_STAMP_RENDER_WAIT) + webrunner_browser.wait_for_captcha_solved(driver, max_wait_seconds=300.0) + + deadline = time.monotonic() + _DOWNLOAD_MAX_WAIT + pdf = _wait_for_new_pdf(out_dir, baseline, deadline) + if pdf is None: + print("[spr] no PDF appeared (likely no institutional access)", flush=True) + return None + return _finalise(pdf, canonical) + + +# --------------------------------------------------------------------------- +# arXiv (open access, no VPN required) +# --------------------------------------------------------------------------- + +_ARXIV_ID_RE = re.compile(r"arxiv\.org/(?:abs|pdf)/(\d{4}\.\d{4,6}(?:v\d+)?)") +_ARXIV_PDF_URL = "https://arxiv.org/pdf/{arxiv_id}.pdf" + + +def arxiv_id_from_url(url: str) -> str | None: + """Pull the arXiv ID out of `arxiv.org/abs/<id>` or `arxiv.org/pdf/<id>`.""" + if not url: + return None + m = _ARXIV_ID_RE.search(url) + return m.group(1) if m else None + + +def download_arxiv(driver: Any, arxiv_id: str, out_dir: Path) -> Path | None: + """Drive Chrome to download an arXiv PDF. arXiv is open-access — no VPN. + + arXiv allows anonymous direct downloads of `/pdf/<id>.pdf`. We use the + same visible-Chrome path as the paywalled publishers (rather than httpx) + so the rest of the batch flow stays consistent — one driver, one log, + same `*.crdownload` polling. + """ + canonical = f"arxiv-{arxiv_id.replace('.', '_').replace('/', '_')}.pdf" + target = out_dir / canonical + if target.exists() and _is_valid_pdf(target): + print(f"[arx] cached {target.name}", flush=True) + return target + _clear_pending(out_dir) + baseline = _snapshot_pdfs(out_dir) + + pdf_url = _ARXIV_PDF_URL.format(arxiv_id=arxiv_id) + print(f"[arx] pdf {pdf_url}", flush=True) + driver.get(pdf_url) + time.sleep(_STAMP_RENDER_WAIT) + + deadline = time.monotonic() + _DOWNLOAD_MAX_WAIT + pdf = _wait_for_new_pdf(out_dir, baseline, deadline) + if pdf is None: + print(f"[arx] no PDF appeared for {arxiv_id}", flush=True) + return None + return _finalise(pdf, canonical) + + +# --------------------------------------------------------------------------- +# ACL Anthology (open access, no VPN required) +# --------------------------------------------------------------------------- + +_ACL_ID_RE = re.compile(r"aclanthology\.org/([^/?#]+?)(?:/|\.pdf)?$") +_ACL_PDF_URL = "https://aclanthology.org/{anthology_id}.pdf" + + +def acl_id_from_url(url: str) -> str | None: + """Pull the ACL Anthology ID out of `aclanthology.org/<id>/`.""" + if not url: + return None + m = _ACL_ID_RE.search(url.rstrip("/") + "/") + if m: + ident = m.group(1) + # Drop trailing `.pdf` / index suffix if matched. + if ident.endswith(".pdf"): + ident = ident[:-4] + return ident + return None + + +def download_aclanthology(driver: Any, anthology_id: str, out_dir: Path) -> Path | None: + """Drive Chrome to download an ACL Anthology PDF. Open access, no VPN.""" + canonical = f"acl-{anthology_id.replace('.', '_').replace('/', '_')}.pdf" + target = out_dir / canonical + if target.exists() and _is_valid_pdf(target): + print(f"[acl] cached {target.name}", flush=True) + return target + _clear_pending(out_dir) + baseline = _snapshot_pdfs(out_dir) + + pdf_url = _ACL_PDF_URL.format(anthology_id=anthology_id) + print(f"[acl] pdf {pdf_url}", flush=True) + driver.get(pdf_url) + time.sleep(_STAMP_RENDER_WAIT) + + deadline = time.monotonic() + _DOWNLOAD_MAX_WAIT + pdf = _wait_for_new_pdf(out_dir, baseline, deadline) + if pdf is None: + print(f"[acl] no PDF appeared for {anthology_id}", flush=True) + return None + return _finalise(pdf, canonical) + + +# --------------------------------------------------------------------------- +# NeurIPS / OpenReview proceedings (open access, no VPN required) +# --------------------------------------------------------------------------- + +_NEURIPS_HASH_RE = re.compile( + r"proceedings\.neurips\.cc/paper(?:_files)?/paper/(\d{4})/hash/([0-9a-f]+)-Abstract" +) + + +def neurips_paper_from_url(url: str) -> tuple[str, str] | None: + """Return ``(year, hash)`` for a NeurIPS proceedings landing URL.""" + if not url: + return None + m = _NEURIPS_HASH_RE.search(url) + return (m.group(1), m.group(2)) if m else None + + +def download_neurips( + driver: Any, year: str, paper_hash: str, out_dir: Path, +) -> Path | None: + """NeurIPS swaps ``hash/<id>-Abstract-Conference.html`` for + ``file/<id>-Paper-Conference.pdf`` to expose the PDF directly.""" + canonical = f"neurips-{year}-{paper_hash[:12]}.pdf" + target = out_dir / canonical + if target.exists() and _is_valid_pdf(target): + print(f"[nips] cached {target.name}", flush=True) + return target + _clear_pending(out_dir) + baseline = _snapshot_pdfs(out_dir) + + pdf_url = ( + f"https://proceedings.neurips.cc/paper_files/paper/{year}/file/" + f"{paper_hash}-Paper-Conference.pdf" + ) + print(f"[nips] pdf {pdf_url}", flush=True) + driver.get(pdf_url) + time.sleep(_STAMP_RENDER_WAIT) + + deadline = time.monotonic() + _DOWNLOAD_MAX_WAIT + pdf = _wait_for_new_pdf(out_dir, baseline, deadline) + if pdf is None: + print(f"[nips] no PDF appeared for {year}/{paper_hash[:12]}", flush=True) + return None + return _finalise(pdf, canonical) + + +_OPENREVIEW_ID_RE = re.compile(r"openreview\.net/(?:forum|pdf)\?id=([A-Za-z0-9]+)") + + +def openreview_id_from_url(url: str) -> str | None: + """Pull the OpenReview paper ID out of `forum?id=…` or `pdf?id=…`.""" + if not url: + return None + m = _OPENREVIEW_ID_RE.search(url) + return m.group(1) if m else None + + +def download_openreview(driver: Any, openreview_id: str, out_dir: Path) -> Path | None: + """OpenReview exposes the PDF at `/pdf?id=<id>` directly. Open access.""" + canonical = f"openreview-{openreview_id}.pdf" + target = out_dir / canonical + if target.exists() and _is_valid_pdf(target): + print(f"[orev] cached {target.name}", flush=True) + return target + _clear_pending(out_dir) + baseline = _snapshot_pdfs(out_dir) + + pdf_url = f"https://openreview.net/pdf?id={openreview_id}" + print(f"[orev] pdf {pdf_url}", flush=True) + driver.get(pdf_url) + time.sleep(_STAMP_RENDER_WAIT) + + deadline = time.monotonic() + _DOWNLOAD_MAX_WAIT + pdf = _wait_for_new_pdf(out_dir, baseline, deadline) + if pdf is None: + print(f"[orev] no PDF appeared for {openreview_id}", flush=True) + return None + return _finalise(pdf, canonical) + + +# --------------------------------------------------------------------------- +# Dispatcher +# --------------------------------------------------------------------------- + +_ACM_DOI_RE = re.compile(r"/doi/(?:abs/|pdf/)?(10\.\d{4,9}/[^\s?#]+)") +_SPRINGER_DOI_RE = re.compile( + r"link\.springer\.com/(?:article|chapter|content/pdf)/(10\.\d{4,9}/[^\s?#]+?)(?:\.pdf)?(?:[?#]|$)" +) + + +def _acm_doi_from_url(url: str) -> str | None: + """Pull a DOI out of any of ACM's URL flavours (`/doi/`, `/doi/abs/`, `/doi/pdf/`).""" + m = _ACM_DOI_RE.search(url) + return m.group(1) if m else None + + +def _springer_doi_from_url(url: str) -> str | None: + """Pull a DOI out of a SpringerLink article/chapter URL.""" + m = _SPRINGER_DOI_RE.search(url) + return m.group(1) if m else None + + +def _doi_prefix_to_publisher(doi: str) -> str | None: + """Map a DOI prefix to a publisher when the URL host is opaque. + + Useful for openalex.org / semanticscholar.org / api.crossref.org URLs + where the host doesn't tell us anything but the DOI does. Covers the + high-volume prefixes — anything else returns None so we don't fake + confidence in publishers we haven't written a downloader for. + """ + prefix = doi.split("/", 1)[0] + return { + "10.1109": "ieee", # IEEE journals + conferences (DOI -> arnumber not directly resolvable; needs search) + "10.1145": "acm", + "10.1007": "springer", # bulk of Springer journals + LNCS + "10.1038": "springer", # Nature family (Springer-published) + }.get(prefix) + + +def _dispatch_paywalled(host: str, url: str, clean_doi: str | None) -> tuple[str, ...] | None: + """VPN-gated publishers — IEEE / ACM / Springer.""" + if "ieeexplore.ieee.org" in host: + arn = arnumber_from_url(url) + return ("ieee", arn) if arn else None + if "dl.acm.org" in host: + resolved = clean_doi or _acm_doi_from_url(url) + return ("acm", resolved) if resolved else None + if "link.springer.com" in host: + resolved = clean_doi or _springer_doi_from_url(url) + return ("springer", resolved) if resolved else None + return None + + +def _dispatch_open_access(host: str, url: str) -> tuple[str, ...] | None: + """Open-access hosts — arXiv, ACL Anthology, NeurIPS, OpenReview.""" + if "arxiv.org" in host: + aid = arxiv_id_from_url(url) + return ("arxiv", aid) if aid else None + if "aclanthology.org" in host: + aid = acl_id_from_url(url) + return ("acl", aid) if aid else None + if "proceedings.neurips.cc" in host: + pair = neurips_paper_from_url(url) + return ("neurips", pair[0], pair[1]) if pair else None + if "openreview.net" in host: + oid = openreview_id_from_url(url) + return ("openreview", oid) if oid else None + return None + + +def _dispatch_by_doi_prefix(clean_doi: str) -> tuple[str, ...] | None: + """Opaque host (openalex / semanticscholar) — pivot to DOI prefix. + + IEEE DOIs (10.1109/...) don't yield an arnumber directly so we return + None — the caller has to resolve via Crossref + IEEE search first. + """ + publisher = _doi_prefix_to_publisher(clean_doi) + if publisher in {"acm", "springer"}: + return (publisher, clean_doi) + return None + + +def dispatch_for_url(url: str, doi: str | None) -> tuple[str, ...] | None: + """Pick the right downloader for a paper's landing URL. + + Returns ``(publisher, *identifier_parts)`` where publisher is one of + ``"ieee" / "acm" / "springer" / "arxiv" / "acl" / "neurips" / + "openreview"`` and the trailing tuple is whatever the matching + ``download_<publisher>`` function expects (arnumber for IEEE, DOI + for ACM / Springer, arXiv ID for arXiv, anthology ID for ACL, + ``(year, hash)`` for NeurIPS, forum ID for OpenReview). Falls back + to extracting the DOI from the URL when the caller-supplied ``doi`` + is empty. Returns ``None`` when the URL is not routable. + """ + if not url: + return None + host = url.split("/", 3)[2].lower() if "://" in url else "" + clean_doi = (doi or "").strip() or None + + paywalled = _dispatch_paywalled(host, url, clean_doi) + if paywalled is not None: + return paywalled + open_access = _dispatch_open_access(host, url) + if open_access is not None: + return open_access + if clean_doi: + return _dispatch_by_doi_prefix(clean_doi) + return None diff --git a/scripts/llm_download_acm_pdf.py b/scripts/llm_download_acm_pdf.py new file mode 100644 index 0000000..1aecf47 --- /dev/null +++ b/scripts/llm_download_acm_pdf.py @@ -0,0 +1,47 @@ +"""LLM-driven ACM Digital Library PDF download (visible Chrome). + +Thin CLI wrapper around `scripts._pdf_downloaders.download_acm`. + +Usage: + .venv\\Scripts\\python.exe -m scripts.llm_download_acm_pdf <doi> + +The DOI is the bare publisher DOI (e.g. ``10.1145/3618257.3624845``), +NOT a full URL. The script navigates to ``https://dl.acm.org/doi/<doi>`` +first to set ACM's session cookies, then to ``/doi/pdf/<doi>`` which +streams the PDF directly when the user has institutional access. Falls +back to iframe-src extraction when ACM wraps the PDF. + +Exit code 0 = PDF saved, 1 = no PDF, 2 = bad arg. + +For a batch over an xlsx, see ``scripts.llm_download_pdfs``. +""" + +from __future__ import annotations + +import contextlib +import sys +from pathlib import Path + +from autopapertoppt.fetchers import webrunner_browser +from scripts._pdf_downloaders import download_acm + +OUT_DIR = Path(r"D:\Codes\AutoPaperToPPT\exports\_llm_scratch\pdfs") +OUT_DIR.mkdir(parents=True, exist_ok=True) + + +def _run(doi: str) -> int: + print(f"[boot] visible Chrome with download_dir={OUT_DIR}", flush=True) + driver = webrunner_browser.make_driver(download_dir=str(OUT_DIR)) + try: + result = download_acm(driver, doi, OUT_DIR) + return 0 if result is not None else 1 + finally: + with contextlib.suppress(Exception): + driver.quit() + + +if __name__ == "__main__": + if len(sys.argv) != 2 or "/" not in sys.argv[1]: + print("usage: python -m scripts.llm_download_acm_pdf <doi>") + sys.exit(2) + sys.exit(_run(sys.argv[1])) diff --git a/scripts/llm_download_ieee_pdf.py b/scripts/llm_download_ieee_pdf.py new file mode 100644 index 0000000..f6213d9 --- /dev/null +++ b/scripts/llm_download_ieee_pdf.py @@ -0,0 +1,42 @@ +"""LLM-driven IEEE PDF download (visible Chrome, no headless). + +Thin CLI wrapper around `scripts._pdf_downloaders.download_ieee`. + +Usage: + .venv\\Scripts\\python.exe -m scripts.llm_download_ieee_pdf <arnumber> + +Output: ``exports/_llm_scratch/pdfs/<arnumber>.pdf`` on success. +Exit code 0 = PDF saved, 1 = no PDF, 2 = bad arg. + +For a batch over an xlsx, see ``scripts.llm_download_pdfs``. +""" + +from __future__ import annotations + +import contextlib +import sys +from pathlib import Path + +from autopapertoppt.fetchers import webrunner_browser +from scripts._pdf_downloaders import download_ieee + +OUT_DIR = Path(r"D:\Codes\AutoPaperToPPT\exports\_llm_scratch\pdfs") +OUT_DIR.mkdir(parents=True, exist_ok=True) + + +def _run(arnumber: str) -> int: + print(f"[boot] visible Chrome with download_dir={OUT_DIR}", flush=True) + driver = webrunner_browser.make_driver(download_dir=str(OUT_DIR)) + try: + result = download_ieee(driver, arnumber, OUT_DIR) + return 0 if result is not None else 1 + finally: + with contextlib.suppress(Exception): + driver.quit() + + +if __name__ == "__main__": + if len(sys.argv) != 2 or not sys.argv[1].isdigit(): + print("usage: python -m scripts.llm_download_ieee_pdf <arnumber>") + sys.exit(2) + sys.exit(_run(sys.argv[1])) diff --git a/scripts/llm_download_pdfs.py b/scripts/llm_download_pdfs.py new file mode 100644 index 0000000..b767e41 --- /dev/null +++ b/scripts/llm_download_pdfs.py @@ -0,0 +1,188 @@ +"""Batch LLM-driven PDF download from an aggregate xlsx. + +Reads a "Papers" sheet produced by ``XlsxExporter`` (columns +``# | Title | Authors | Year | Source | Indexed via | DOI | URL | +PDF | Citations | Abstract``), groups rows by publisher, opens a +SINGLE visible Chrome session, and walks each paper in turn. Reusing +one driver across N papers means: cookies / VPN auth survive, captcha +(if any) is solved once, and the per-paper overhead drops to one +``driver.get`` + a download wait. + +Supported publishers (URL host → handler): +* ``ieeexplore.ieee.org`` → ``download_ieee`` (uses arnumber from URL) +* ``dl.acm.org`` → ``download_acm`` (uses DOI) +* ``link.springer.com`` → ``download_springer`` (uses DOI) + +Rows whose URL host isn't in the table above are skipped with a note. + +Usage: + .venv\\Scripts\\python.exe -m scripts.llm_download_pdfs <xlsx_path> + .venv\\Scripts\\python.exe -m scripts.llm_download_pdfs <xlsx_path> \\ + --publishers ieee,acm + +Outputs land in ``exports/_llm_scratch/pdfs/`` next to the per-paper +downloads from the single-paper CLIs. Exit code: 0 when every paper +downloaded, 1 when one or more failed. +""" + +from __future__ import annotations + +import argparse +import contextlib +import sys +from pathlib import Path +from typing import Any + +from openpyxl import load_workbook + +from autopapertoppt.fetchers import webrunner_browser +from scripts._pdf_downloaders import ( + dispatch_for_url, + download_aclanthology, + download_acm, + download_arxiv, + download_ieee, + download_neurips, + download_openreview, + download_springer, +) + +OUT_DIR = Path(r"D:\Codes\AutoPaperToPPT\exports\_llm_scratch\pdfs") +OUT_DIR.mkdir(parents=True, exist_ok=True) + + +def _invoke(publisher: str, args: tuple, driver: Any, out_dir: Path) -> Path | None: + """Route ``(publisher, *args)`` to its downloader.""" + if publisher == "ieee": + return download_ieee(driver, args[0], out_dir) + if publisher == "acm": + return download_acm(driver, args[0], out_dir) + if publisher == "springer": + return download_springer(driver, args[0], out_dir) + if publisher == "arxiv": + return download_arxiv(driver, args[0], out_dir) + if publisher == "acl": + return download_aclanthology(driver, args[0], out_dir) + if publisher == "neurips": + return download_neurips(driver, args[0], args[1], out_dir) + if publisher == "openreview": + return download_openreview(driver, args[0], out_dir) + raise ValueError(f"unknown publisher: {publisher!r}") + + +def _load_papers(xlsx_path: Path) -> list[dict[str, str]]: + """Return one dict per row in the 'Papers' sheet.""" + wb = load_workbook(xlsx_path, read_only=True, data_only=True) + sheet = wb["Papers"] + rows = list(sheet.iter_rows(values_only=True)) + if not rows: + return [] + headers = [str(h or "").strip() for h in rows[0]] + out: list[dict[str, str]] = [] + for row in rows[1:]: + record: dict[str, str] = {} + for header, value in zip(headers, row, strict=False): + record[header] = "" if value is None else str(value) + out.append(record) + return out + + +def _plan( + papers: list[dict[str, str]], publisher_filter: set[str], +) -> list[tuple[str, tuple[str, ...], str]]: + """Pick rows we can download. Returns ``(publisher, identifier_parts, title_preview)``. + + ``identifier_parts`` is a tuple so multi-part identifiers (NeurIPS uses + ``(year, hash)``) ride through the planner without special casing. + """ + plan: list[tuple[str, tuple[str, ...], str]] = [] + seen: set[tuple] = set() + for row in papers: + url = row.get("URL", "") or row.get("Url", "") + doi = row.get("DOI", "") or row.get("Doi", "") + dispatch = dispatch_for_url(url, doi or None) + if dispatch is None: + continue + publisher, *identifier_parts = dispatch + ident_tuple = tuple(identifier_parts) + if publisher_filter and publisher not in publisher_filter: + continue + key = (publisher, *ident_tuple) + if key in seen: + continue + seen.add(key) + title = (row.get("Title") or "")[:60] + plan.append((publisher, ident_tuple, title)) + return plan + + +def _run(xlsx_path: Path, publisher_filter: set[str]) -> int: + papers = _load_papers(xlsx_path) + plan = _plan(papers, publisher_filter) + if not plan: + print(f"[plan] nothing to download from {xlsx_path}", flush=True) + return 0 + + by_pub: dict[str, list[tuple[str, str]]] = {} + for publisher, ident_tuple, title in plan: + by_pub.setdefault(publisher, []).append(("/".join(ident_tuple), title)) + print( + "[plan] " + ", ".join( + f"{p}={len(rows)}" for p, rows in sorted(by_pub.items()) + ), + flush=True, + ) + + print(f"[boot] visible Chrome with download_dir={OUT_DIR}", flush=True) + driver = webrunner_browser.make_driver(download_dir=str(OUT_DIR)) + failures: list[tuple[str, str, str]] = [] + successes: list[tuple[str, str, Path]] = [] + try: + for publisher, ident_tuple, title in plan: + ident_str = "/".join(ident_tuple) + print( + f"\n=== {publisher} :: {ident_str} :: {title!r} ===", + flush=True, + ) + try: + saved = _invoke(publisher, ident_tuple, driver, OUT_DIR) + except Exception as err: # noqa: BLE001 — selenium raises many types + print(f"[err] {publisher} {ident_str} raised: {err}", flush=True) + failures.append((publisher, ident_str, f"exception: {err}")) + continue + if saved is None: + failures.append((publisher, ident_str, "no PDF produced")) + else: + successes.append((publisher, ident_str, saved)) + finally: + with contextlib.suppress(Exception): + driver.quit() + + print("\n=== summary ===") + print(f" ok={len(successes)} fail={len(failures)} total={len(plan)}") + for pub, ident, path in successes: + print(f" [ok] {pub} {ident} -> {path.name} ({path.stat().st_size:,} bytes)") + for pub, ident, reason in failures: + print(f" [fail] {pub} {ident} :: {reason}") + return 0 if not failures else 1 + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + parser.add_argument("xlsx", help="path to an XlsxExporter 'Papers' sheet") + parser.add_argument( + "--publishers", + default="", + help="comma-separated subset of {ieee,acm,springer}; default = all", + ) + args = parser.parse_args() + xlsx_path = Path(args.xlsx) + if not xlsx_path.is_file(): + print(f"[err] xlsx not found: {xlsx_path}") + return 2 + publisher_filter = {p.strip() for p in args.publishers.split(",") if p.strip()} + return _run(xlsx_path, publisher_filter) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/llm_download_springer_pdf.py b/scripts/llm_download_springer_pdf.py new file mode 100644 index 0000000..40dfc83 --- /dev/null +++ b/scripts/llm_download_springer_pdf.py @@ -0,0 +1,47 @@ +"""LLM-driven SpringerLink PDF download (visible Chrome). + +Thin CLI wrapper around `scripts._pdf_downloaders.download_springer`. + +Usage: + .venv\\Scripts\\python.exe -m scripts.llm_download_springer_pdf <doi> + +The DOI is the bare publisher DOI (e.g. ``10.1007/978-981-96-1024-2_8``). +The script tries ``/article/<doi>`` first, falls back to +``/chapter/<doi>`` when the article path 404s (book chapters live under +``/chapter/``), then navigates to ``/content/pdf/<doi>.pdf`` which +streams the PDF when the user's network has institutional access. + +Exit code 0 = PDF saved, 1 = no PDF, 2 = bad arg. + +For a batch over an xlsx, see ``scripts.llm_download_pdfs``. +""" + +from __future__ import annotations + +import contextlib +import sys +from pathlib import Path + +from autopapertoppt.fetchers import webrunner_browser +from scripts._pdf_downloaders import download_springer + +OUT_DIR = Path(r"D:\Codes\AutoPaperToPPT\exports\_llm_scratch\pdfs") +OUT_DIR.mkdir(parents=True, exist_ok=True) + + +def _run(doi: str) -> int: + print(f"[boot] visible Chrome with download_dir={OUT_DIR}", flush=True) + driver = webrunner_browser.make_driver(download_dir=str(OUT_DIR)) + try: + result = download_springer(driver, doi, OUT_DIR) + return 0 if result is not None else 1 + finally: + with contextlib.suppress(Exception): + driver.quit() + + +if __name__ == "__main__": + if len(sys.argv) != 2 or "/" not in sys.argv[1]: + print("usage: python -m scripts.llm_download_springer_pdf <doi>") + sys.exit(2) + sys.exit(_run(sys.argv[1])) diff --git a/scripts/llm_driven_search.py b/scripts/llm_driven_search.py new file mode 100644 index 0000000..daf9389 --- /dev/null +++ b/scripts/llm_driven_search.py @@ -0,0 +1,109 @@ +"""LLM-driven search: visible Chrome, the LLM picks URLs. + +The CLI's built-in WebRunner backend (`sources/ieee/webrunner_backend.py`, +`sources/scholar/webrunner_backend.py`) is the *Python pipeline* path — +it boots Chrome from inside `asyncio.gather`, captures HTML/JSON, and +hands it to the parsers. That works for unattended CI but burns the +LLM's ability to make per-step decisions (which paper to dig into, +which page to scroll, when to give up on a captcha). + +This script is the *LLM-as-agent* path: the LLM in a Claude Code session +invokes this script via Bash, the script opens a visible Chrome window +(no headless), navigates to Scholar + IEEE for a chosen query, captures +the SERP HTML and `/rest/search` JSON to disk, and quits. The LLM then +calls `llm_parse_results.py` to merge / dedup / rank / export. + +The split exists because Selenium sessions don't survive across Bash +invocations — once Chrome quits, state is gone. Keeping capture and +parse in separate scripts means the LLM can inspect each capture +(via the Read tool on the dumped HTML/JSON) before deciding next steps, +e.g. "the SERP returned a captcha, ask the user to solve it and re-run." + +Usage: + .venv\\Scripts\\python.exe -m scripts.llm_driven_search "your query" + +Output: ``exports/_llm_scratch/scholar.html`` and +``exports/_llm_scratch/ieee_search.json``. +""" + +from __future__ import annotations + +import contextlib +import json +import sys +import time +from pathlib import Path + +from autopapertoppt.fetchers import webrunner_browser + +OUT_DIR = Path(r"D:\Codes\AutoPaperToPPT\exports\_llm_scratch") +OUT_DIR.mkdir(parents=True, exist_ok=True) + + +def _drive(query: str) -> None: + print("[boot] launching visible Chrome ...", flush=True) + driver = webrunner_browser.make_driver() + try: + # ---- Scholar ---- + scholar_url = ( + "https://scholar.google.com/scholar" + f"?q={query.replace(' ', '+')}&hl=en&num=10" + ) + print(f"[scholar] navigate {scholar_url}", flush=True) + driver.get(scholar_url) + time.sleep(4) + webrunner_browser.wait_for_captcha_solved(driver, max_wait_seconds=300.0) + scholar_html_path = OUT_DIR / "scholar.html" + scholar_html_path.write_text(driver.page_source, encoding="utf-8") + print( + f"[scholar] page_source bytes={len(driver.page_source)} " + f"-> {scholar_html_path}", + flush=True, + ) + + # ---- IEEE Xplore: land on home so REST cookies are set, + # then JS-fetch the /rest/search endpoint from the IEEE origin. + print("[ieee] navigate https://ieeexplore.ieee.org/Xplore/home.jsp", flush=True) + driver.get("https://ieeexplore.ieee.org/Xplore/home.jsp") + time.sleep(4) + webrunner_browser.wait_for_captcha_solved(driver, max_wait_seconds=300.0) + driver.set_script_timeout(30) + rest_body = { + "queryText": query, + "highlight": False, + "returnFacets": ["ALL"], + "returnType": "SEARCH", + "matchPubs": True, + "pageNumber": 1, + "rowsPerPage": 10, + } + js = ( + "const url=arguments[0], body=arguments[1], cb=arguments[2];" + "fetch(url,{method:'POST',headers:{" + "'Accept':'application/json,text/plain,*/*'," + "'Content-Type':'application/json'," + "'Origin':'https://ieeexplore.ieee.org'," + "'Referer':'https://ieeexplore.ieee.org/search/searchresult.jsp'" + "},credentials:'include',body:body})" + ".then(r=>r.json()).then(j=>cb(j))" + ".catch(e=>cb({_error:String(e)}));" + ) + result = driver.execute_async_script( + js, "https://ieeexplore.ieee.org/rest/search", json.dumps(rest_body) + ) + ieee_json_path = OUT_DIR / "ieee_search.json" + ieee_json_path.write_text(json.dumps(result, indent=2), encoding="utf-8") + print( + f"[ieee] records={len((result or {}).get('records') or [])} " + f"-> {ieee_json_path}", + flush=True, + ) + finally: + with contextlib.suppress(Exception): + driver.quit() + print("[done] chrome quit", flush=True) + + +if __name__ == "__main__": + q = sys.argv[1] if len(sys.argv) > 1 else "test-time compute scaling reasoning LLM" + _drive(q) diff --git a/scripts/llm_parse_results.py b/scripts/llm_parse_results.py new file mode 100644 index 0000000..7f3f2dc --- /dev/null +++ b/scripts/llm_parse_results.py @@ -0,0 +1,80 @@ +"""Parse the artefacts left by _llm_driven_search.py. + +Run after _llm_driven_search.py finishes. Reads the captured Scholar +HTML + IEEE JSON, runs the project's parsers, merges + de-dupes via the +existing core helpers, and writes a small markdown + xlsx the LLM can +hand the user. +""" + +from __future__ import annotations + +import json +import sys +from pathlib import Path + +# Mirror the runtime injection that autopapertoppt/fetchers/base.py performs +# at load time so `from ieee.parser import ...` resolves. +_SOURCES = Path(__file__).resolve().parents[1] / "sources" +if str(_SOURCES) not in sys.path: + sys.path.insert(0, str(_SOURCES)) + +from ieee.parser import parse_search_record # noqa: E402 # sources/ path injection above +from scholar.parser import parse_serp # noqa: E402 # sources/ path injection above + +from autopapertoppt.core.dedup import dedupe # noqa: E402 # after sys.path injection +from autopapertoppt.core.models import ExportOptions, PaperCollection, Query # noqa: E402 +from autopapertoppt.core.ranking import rank # noqa: E402 +from autopapertoppt.exporters.markdown import MarkdownExporter # noqa: E402 +from autopapertoppt.exporters.xlsx import XlsxExporter # noqa: E402 + +ROOT = Path(r"D:\Codes\AutoPaperToPPT\exports\_llm_scratch") +QUERY_STR = "test-time compute scaling reasoning LLM" + + +def _load_scholar() -> list: + html = (ROOT / "scholar.html").read_text(encoding="utf-8") + return parse_serp(html) + + +def _load_ieee() -> list: + data = json.loads((ROOT / "ieee_search.json").read_text(encoding="utf-8")) + return [parse_search_record(r) for r in (data.get("records") or [])] + + +def main() -> None: + scholar_papers = _load_scholar() + ieee_papers = _load_ieee() + print(f"scholar parsed: {len(scholar_papers)}") + print(f"ieee parsed: {len(ieee_papers)}") + + merged = dedupe(scholar_papers + ieee_papers) + ranked = rank(merged) + print(f"after dedup+rank: {len(ranked)}") + + query = Query( + keywords=QUERY_STR, + max_results=25, + sources=("scholar", "ieee"), + ) + collection = PaperCollection(query=query, papers=tuple(ranked[:25])) + + options = ExportOptions( + formats=("xlsx", "md"), + out_dir=str(ROOT), + filename_stem="llm_driven", + include_abstract=True, + ) + xlsx_path = XlsxExporter().export(collection, options) + md_path = MarkdownExporter().export(collection, options) + print(f"xlsx: {xlsx_path}") + print(f"md: {md_path}") + + print("\n--- Top 10 ---") + for i, p in enumerate(ranked[:10], 1): + title = (p.title or "")[:78] + via = p.source or "?" + print(f" [{i:>2}] ({p.year}) {title} [via {via}]") + + +if __name__ == "__main__": + main() diff --git a/scripts/regen_speculative_decoding_zh_tw.py b/scripts/regen_speculative_decoding_zh_tw.py new file mode 100644 index 0000000..81a0b80 --- /dev/null +++ b/scripts/regen_speculative_decoding_zh_tw.py @@ -0,0 +1,804 @@ +"""Traditional Chinese (zh-tw) rich decks for 4 speculative-decoding papers. + +Built via the LLM-as-agent path: PDFs downloaded by +scripts/llm_download_pdfs.py (extended dispatcher handles +arXiv / ACL / NeurIPS / IEEE), then this script bundles a +hand-authored rich PaperSummary per paper and exports one +``<key>-zh-tw.pptx`` per paper. + +The 5th xlsx row (OpenAlex W4405717632) is an OpenAlex wrapper of the +same EdgeLLM paper as row 3; it cannot be downloaded directly (IEEE +DOI does not yield an arnumber), so it is consciously skipped here. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) +sys.path.insert(0, str(ROOT / "sources")) + +from autopapertoppt.core.models import ( # noqa: E402 + ExportOptions, + Paper, + PaperCollection, + PaperSummary, + Query, + RqResult, +) +from autopapertoppt.exporters import export_collection # noqa: E402 + +MODEL_TAG = "LLM-as-agent (讀完整 PDF)" +_RUN_DIR_NAME = sys.argv[1] if len(sys.argv) > 1 else "speculative-decoding-zh-tw" + + +# --------------------------------------------------------------------------- +# 1. Xia et al. 2024 — Speculative Decoding Survey (ACL Findings) +# --------------------------------------------------------------------------- +XIA = Paper( + source="local", + source_id="xia2024speculative", + title="Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding", + authors=( + "Heming Xia", "Zhe Yang", "Qingxiu Dong", "Peiyi Wang", + "Yongqi Li", "Tao Ge", "Tianyu Liu", "Wenjie Li", "Zhifang Sui", + ), + year=2024, + venue="ACL 2024 Findings", + abstract="第一篇系統性綜述 Speculative Decoding 的論文,提供形式化定義、新的分類體系,並推出 Spec-Bench 統一基準,讓未來研究有共同比較基礎。", + url="https://aclanthology.org/2024.findings-acl.456/", + doi=None, + pdf_url=None, + summary=PaperSummary( + language="zh-tw", + model=MODEL_TAG, + raw_text_chars=73_248, + pain_points=( + ("自回歸解碼是 LLM 推論的硬瓶頸", ( + "每步只生成一個 token,GPU 利用率極低", + "瓶頸不在算力而在 HBM → 晶片 cache 的搬運", + "模型越大、memory-bound 的代價越明顯", + )), + ("Speculative Decoding 散見於文獻、缺整合視角", ( + "Blockwise、SpecDec、SpecSampling 各自演化", + "drafter 設計、verify 策略命名不一", + "新手難以快速進入領域", + )), + ("各方法的 speedup 數字無法直接比較", ( + "硬體、模型、batch、prompt 都不一樣", + "缺少 third-party 統一測試環境", + "業界引用時容易誤解 speedup 的可移植性", + )), + ("Drafter 設計面臨投機準確度 vs 延遲的拉扯", ( + "Drafter 太小 → 接受率低、無 speedup", + "Drafter 太大 → 自身延遲吃掉好處", + "如何 align 兩個模型的行為是公開問題", + )), + ), + research_question=( + "如何系統性整理 Speculative Decoding 研究," + "並提供統一基準讓未來方法能在同樣條件下被公平比較?" + ), + contributions_detailed=( + ("一、首篇 Speculative Decoding 綜述", + "整合 2018 Blockwise 以來所有方向,把 Draft-then-Verify 抬升為一個獨立解碼範式。"), + ("二、形式化定義與演算法", + "提供 Algorithm 2 的標準寫法,把 DRAFT / VERIFY / CORRECT 三個子程序明確化。"), + ("三、新的分類體系", + "Drafting (Independent / Self) × Verification (Greedy / SpecSampling / Token-Tree) 雙軸分類,涵蓋現有 20+ 方法。"), + ("四、Spec-Bench 統一基準", + "跨應用場景的標準測試環境,讓不同方法在同硬體 / 模型 / prompt 下比較。"), + ), + headline_metrics=( + ("SpecDec speedup", "≈5×", "vs 標準自回歸解碼 (Xia et al. 2023)"), + ("分類涵蓋的代表方法數", "20+", "Drafting 12 + Verification 8 條"), + ("Drafting 分支類別", "2", "Independent (tuning-free/fine-tuned) + Self-Drafting"), + ("Verification 分支類別", "3", "Greedy / Speculative Sampling / Token Tree"), + ("Spec-Bench 涵蓋場景", "6", "多輪對話 / 翻譯 / 摘要 / 問答 / 數學 / 推理"), + ), + technique_table=( + ("Draft-then-Verify 範式", "起源於 Stern 2018 的 Blockwise Decoding"), + ("Independent Drafter", "外部小模型 (T5-small、GPT-2 small) 充當 drafter"), + ("Self-Drafter (FFN heads)", "Medusa / Blockwise — 在原 LLM 上加 head 並行出 token"), + ("Self-Drafter (Early Exit)", "Self-Speculative / SPEED — 提早離開若干層作 draft"), + ("Token-Tree Verification", "SpecInfer / Medusa — 同時驗證樹狀分支提升接受率"), + ("Knowledge Distillation", "DistillSpec — 訓練 drafter 對齊 target 的輸出分佈"), + ), + method_sections=( + ("Drafting (§5)", ( + "Independent Drafting:外部小模型或 NAR Transformer", + "Self-Drafting:FFN heads / early exit / mask-predict", + "兩條路線各自有 tuning-free 與 fine-tuned 變體", + )), + ("Verification (§6)", ( + "Greedy Decoding 標準 + 近似", + "Speculative Sampling 維持原 distribution 的接受規則", + "Token Tree Verification 並行驗證多分支", + )), + ), + evaluation_sections=( + ("Spec-Bench 評測設定", ( + "Vicuna-7B 為 target,多種 drafter 變體", + "MT-Bench / CNN-DM / WMT / CoT 等 6 種任務", + "single-GPU 標準硬體 (A100 80GB)", + )), + ("比較指標", ( + "Mean Accepted Tokens (MAT) 評接受率", + "Wall-clock speedup vs 純自回歸 baseline", + "FLOPs / memory bandwidth 拆解分析", + )), + ), + system_flow=( + "輸入 prompt 進入 target LLM", + "Drafter Mp 並行/自回歸生成 K 個草稿 token", + "Target Mq 一次 forward 同時驗證 K+1 個分佈", + "依 VERIFY 條件接受或在第一個 mismatch 處 CORRECT", + "若全部接受則額外從 q_{K+1} 取一個 token", + "迴圈直到 [EOS] 或長度上限", + ), + research_questions=( + ("RQ1", "如何設計兼顧投機準確度與延遲的 drafter?"), + ("RQ2", "verify 策略如何在 quality / parallelism 之間取捨?"), + ("RQ3", "drafter 與 target 的行為對齊有哪些可行做法?"), + ("RQ4", "Speculative Decoding 在多任務多硬體下的真實 speedup?"), + ), + rq_results=( + RqResult( + rq_id="RQ1", + question="獨立 vs 自我 drafter 的權衡", + table=( + ("Drafter 類型", "代表方法", "優點", "缺點"), + ("Independent (off-the-shelf)", "Spec Sampling, StagedSpec", "免訓練", "需有同系列小模型"), + ("Independent (fine-tuned)", "SpecDec, BiLD", "對齊度高", "需額外訓練資料"), + ("Self (FFN heads)", "Medusa, EAGLE", "免外部模型", "需修改架構"), + ("Self (Early Exit)", "Self-Speculative, SPEED", "完全免訓練", "early exit 品質下降"), + ), + analysis=( + "Self-drafter 在分散式部署更友善", + "Independent 在 7B–70B 模型上 speedup 較穩", + "Trade-off 仍取決於目標硬體與 batch 大小", + ), + ), + RqResult( + rq_id="RQ2", + question="不同 verification 策略的取捨", + table=( + ("策略", "Quality", "並行度", "代表方法"), + ("Greedy + lossless", "完全保真", "中", "Blockwise, SpecDec"), + ("Greedy + approximate", "略有偏差", "高", "BiLD rollback"), + ("Spec Sampling", "保真 (隨機)", "中", "Leviathan, SpS"), + ("Token Tree", "完全保真", "極高", "SpecInfer, Medusa, EAGLE"), + ), + analysis=( + "Token Tree 在大 batch 提升最顯著", + "Approximate greedy 用於可容忍誤差場景", + "Spec Sampling 是隨機解碼的標準選擇", + ), + ), + RqResult( + rq_id="RQ3", + question="Drafter–Target 對齊技術", + table=( + ("對齊技術", "方法"), + ("Knowledge Distillation", "DistillSpec — 蒸餾 target 的 logits"), + ("Online Adaptation", "Online Speculative — 線上更新 drafter"), + ("結構共享", "Medusa head 從 target 自身衍生"), + ("Tokenizer 對齊", "同系列模型自然對齊"), + ), + analysis=( + "Distillation 在初次 deploy 時對齊度最好", + "Online 對齊適合 distribution shift 場景", + "結構共享避免雙模型部署成本", + ), + ), + ), + core_observation=( + "Speculative Decoding 的 speedup 取決於 drafted token 的" + "接受率,而接受率被 drafter 設計、verify 條件、與 target 的" + "行為對齊三條軸線共同決定。Spec-Bench 把這三條軸線的影響" + "量化,讓未來方法可以針對最弱的環節改良。" + ), + limitations=( + "Spec-Bench 仍以單 GPU 為主,多 GPU / 多節點 setting 待擴", + "綜述截至 2024 上半年,後續方法 (Eagle-2、Hydra) 未納入", + "Token Tree 在極長 context 的記憶體成本尚無系統分析", + "Drafter 線上更新對 production 部署的代價待量化", + ), + future_work=( + "更廣硬體 (mobile / edge) 上的 Speculative Decoding 評估", + "結合 quantization / KV-cache 壓縮的協同最佳化", + "多模態 LLM 的 Speculative Decoding 變體", + "Drafter 的自適應 / 持續學習機制", + ), + ), +) + + +# --------------------------------------------------------------------------- +# 2. Spector & Re 2023 — Staged Speculative Decoding (ICML) +# --------------------------------------------------------------------------- +SPECTOR = Paper( + source="local", + source_id="spector2023staged", + title="Accelerating LLM Inference with Staged Speculative Decoding", + authors=("Benjamin Spector", "Chris Re"), + year=2023, + venue="ICML 2023 ES-FoMo Workshop", + abstract="提出 staged speculative decoding,把 speculative batch 重構成樹並加入第二層 draft 模型 (N-gram),在 GPT-2-L 上達 3.16× 加速且完全保真。", + url="https://arxiv.org/abs/2308.04623", + doi=None, + arxiv_id="2308.04623", + pdf_url=None, + summary=PaperSummary( + language="zh-tw", + model=MODEL_TAG, + raw_text_chars=22_837, + pain_points=( + ("Small-batch on-device 推論 arithmetic intensity 低", ( + "16-bit、batch=1 時 AI 僅約 1", + "RTX 4090 在 GPT-2-L 只跑 150 t/s,僅 0.13% 利用率", + "受限於 memory bandwidth 的 roofline", + )), + ("標準 Speculative Decoding 飽和快", ( + "Drafter 連續預測正確的機率指數下降", + "增大 speculative width 反而拖垮 drafter 自身", + "draft cost 在大 batch 反客為主", + )), + ("Cloud 推論不總是可行", ( + "低延遲應用 (即時對話) 雲端不夠快", + "隱私敏感資料不能離開設備", + "個人化模型適合 local 微調", + )), + ("Drafter 大小是難以調的超參", ( + "太大 → align 好但成本高", + "太小 → 接受率低、速度反而下降", + "經驗值 15-20× 縮小但仍不是 free lunch", + )), + ), + research_question=( + "在 small-batch on-device 場景下,如何進一步打破 Speculative " + "Decoding 的飽和上限,同時完全保留 model 輸出分佈?" + ), + contributions_detailed=( + ("一、樹狀 Speculative Batch", + "把原本單一序列改成可能 token 的樹,提升 expected tokens/batch、增加 leaf 數量、且 drafter 只在內部節點執行。"), + ("二、第二層 Draft (Staged)", + "在 GPT-2 40M draft 之下再加一個 Katz N-gram 模型作 draft2,讓 drafter 自身也享受 speculative 加速。"), + ("三、3.16× wall-clock speedup", + "RTX 4090 + GPT-2-L 762M oracle,deterministic decoding 從 150 t/s 推到 475 t/s,完全保真。"), + ("四、低 entropy token 的觀察", + "多數 token 熵低 (空白、縮排) 可由 N-gram 即時供給,只有少數關鍵 token 才必須走 oracle。"), + ), + headline_metrics=( + ("Deterministic 解碼吞吐", "475 t/s", "baseline 150 / spec 350 (3.16× / 1.36×)"), + ("Topk 解碼吞吐", "298 t/s", "baseline 150 / spec 219 (1.98× / 1.36×)"), + ("Memory bandwidth 比例", "0.23", "baseline 1.00 / spec 0.31 (deterministic)"), + ("Oracle 模型", "GPT-2-L 762M", "fine-tuned on The Stack Python"), + ("Draft 模型", "GPT-2 40M", "20× 小於 oracle"), + ("Draft2 模型", "Katz N-gram", "由 draft 跑 2 小時生成 120M token 訓"), + ), + technique_table=( + ("Tree-structured batch", "把線性序列改為 token tree,擴張 leaf"), + ("KV-cache 切分", "self-attention 拆成 cross-attn + batch-internal self-attn"), + ("Causal masking on tree", "依樹結構控制 positional embed 與 attention mask"), + ("3-tier hierarchy", "Oracle (762M) → Draft (40M) → Draft2 (N-gram)"), + ("Rejection sampling", "對 topk 採用,保證最終分佈與 oracle 相同"), + ("HumanEval 評測語料", "164 個 prompt,涵蓋 Python 程式碼生成"), + ), + method_sections=( + ("Tree-structured Speculative Batch (§3.1)", ( + "在 root 之下動態長出多分支樹,涵蓋 top-k token", + "Drafter 只在內部節點 forward 一次,葉子 free", + "KV cache 為整樹獨立儲存,接受後再 append 主 cache", + )), + ("Staged Speculation (§3.2)", ( + "Katz N-gram 對 draft 自身做 speculative", + "Drafter 自己也擁有更小的 draft → speedup 累乘", + "三層 hierarchy 透過 rejection sampling 維持分佈", + )), + ), + evaluation_sections=( + ("Bandwidth 量測", ( + "標 baseline / spec / staged 三組", + "Deterministic: 1.00 / 0.31 / 0.23", + "Topk: 1.00 / 0.48 / 0.35", + )), + ("吞吐量量測", ( + "HumanEval 164 個 prompt 平均", + "Deterministic + Topk(k=50, T=1) 兩種", + "Profiling 顯示 35% 來自 Python 開銷", + )), + ), + system_flow=( + "Prompt 進入 GPT-2-L oracle 取得 KV cache + 首 logit", + "N-gram 在 < 10µs 內預測 top-k token 形成樹的根層", + "GPT-2 40M draft 在內部節點 forward 補足更深層", + "Tree-shaped batch 一次送入 oracle 驗證", + "通過驗證的分支接受、未通過處退回 oracle 取代", + "迴圈直到 [EOS] 或長度上限", + ), + research_questions=( + ("RQ1", "樹狀 batch 是否比單序列 speculative 有效?"), + ("RQ2", "對 draft 自身再做 speculative 是否有額外加速?"), + ("RQ3", "整體方法是否完全保留 model 分佈?"), + ), + rq_results=( + RqResult( + rq_id="RQ1", + question="Tree batch vs 單序列 speculative", + table=( + ("方法", "Det. bandwidth", "Topk bandwidth", "備註"), + ("Baseline (no spec)", "1.00", "1.00", "純自回歸"), + ("Standard Speculative", "0.31", "0.48", "Leviathan 2022"), + ("Staged (tree-only ablation)", "≈0.28", "≈0.40", "去掉 draft2 後估算"), + ("Staged (full)", "0.23", "0.35", "tree + draft2"), + ), + analysis=( + "Tree batch 本身已壓低 bandwidth", + "Tree 帶來更多 free leaf token", + "Drafter 在內部節點才需 forward", + ), + ), + RqResult( + rq_id="RQ2", + question="Draft2 (N-gram) 的邊際貢獻", + table=( + ("解碼模式", "Spec t/s", "Staged t/s", "額外 speedup"), + ("Deterministic", "350", "475", "1.36×"), + ("Topk (k=50, T=1)", "219", "298", "1.36×"), + ), + analysis=( + "Draft2 處理低熵 token (空白、縮排)", + "對 draft 自身的 forward 次數降低", + "Drafter 自身在小 batch 也是 bandwidth-bound", + ), + ), + RqResult( + rq_id="RQ3", + question="分佈保真性", + table=( + ("項目", "結果"), + ("Deterministic 輸出", "與 oracle bit-exact"), + ("Topk 輸出 (rejection sampling)", "分佈與 oracle 相同"), + ("HumanEval pass@1", "與 oracle 一致"), + ), + analysis=( + "Rejection sampling 保證機率正確", + "Wall-clock 加速不換取品質下降", + "與 quantization 等技術正交", + ), + ), + ), + core_observation=( + "多數 LLM 輸出的 token 熵低,可由極輕量模型 (甚至 N-gram) 即時" + "供給;只有少數關鍵 token 才必須走完整 oracle。把這個觀察" + "操作化為樹狀 + 多階段 draft 之後,RTX 4090 上 GPT-2-L 從 150 " + "推到 475 t/s 且分佈不變。意味著推論成本與生成文字的 entropy " + "本身綁在一起,而非與模型大小直接成正比。" + ), + limitations=( + "35% 來自 Python infrastructure,C++/CUDA 化會更快", + "Speedup 隨 prompt 內容變化大 (2× ~ 10×)", + "只在 762M 上驗證,真實大模型行為待驗", + "Drafter 為同領域 fine-tuned,跨領域 align 度未知", + ), + future_work=( + "T>0 sampling 可先採 multinomial CDF 再選 batch token", + "8-bit quant 後可在消費 GPU 跑 20B → 1B → 50M → N-gram 四階段", + "更好的 lowest-level drafter (<10µs 但勝於 N-gram)", + "與 quantization、Flash-Attn 等技術的協同最佳化", + ), + ), +) + + +# --------------------------------------------------------------------------- +# 3. Xu et al. 2024 — EdgeLLM (IEEE TMC) +# --------------------------------------------------------------------------- +XU_EDGELLM = Paper( + source="local", + source_id="xu2024edgellm", + title="EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding", + authors=( + "Daliang Xu", "Wangsong Yin", "Hao Zhang", "Xin Jin", + "Ying Zhang", "Shiyun Wei", "Mengwei Xu", "Xuanzhe Liu", + ), + year=2024, + venue="IEEE Transactions on Mobile Computing 24(4)", + abstract="在 mobile / IoT 上用 speculative decoding 突破裝置記憶體上限,三項新技巧 (寬度自適應 token tree、自適應 fallback、provisional generation pipeline) 帶來 2.9–9.3× 加速。", + url="https://ieeexplore.ieee.org/abstract/document/10812936/", + doi=None, + pdf_url=None, + summary=PaperSummary( + language="zh-tw", + model=MODEL_TAG, + raw_text_chars=93_090, + pain_points=( + ("Mobile LLM 撞上 memory wall", ( + "10B 是 Xiaomi 10 能即時的上限", + "超出記憶體 → MNN / llama.cpp 反覆從 disk 換入 weight", + "推論延遲拉長 59–224×", + )), + ("Speculative Decoding 在 edge 有新挑戰", ( + "Token tree 增寬會壓垮資源受限的 drafter", + "Verification 時機難判定;早 / 晚都浪費", + "Verify 期間 draft 必須暫停,I/O 與 compute 不對稱", + )), + ("既有 mobile DNN engine 對 LLM 不友善", ( + "MNN / llama.cpp 走 swap 策略", + "Disk I/O 占推論時間 95.9–98.8%", + "Batching / pipeline 對 autoregressive 失效", + )), + ("壓縮 / sparsity 方法犧牲精度", ( + "Quantization 在極小裝置上仍會掉點", + "Context sparsity 在多輪對話崩潰", + "需要不犧牲精度的加速路線", + )), + ), + research_question=( + "如何在記憶體不足以裝載目標 LLM 的 mobile / IoT 裝置上," + "用 speculative decoding 兼顧記憶體上限與不損失精度的加速?" + ), + contributions_detailed=( + ("一、寬度自適應 Token Tree 與批次驗證", + "依 token / branch confidence 動態調整每分支寬度,並一次 batch 驗證整棵樹,把 oracle 呼叫降至最低。"), + ("二、Self-Adaptive Fallback 策略", + "結合候選分支 joint confidence 與歷史 verify 準確度,動態調整 fallback 觸發門檻。"), + ("三、Provisional Generation Pipeline", + "Verify 期間 drafter 不暫停,持續預生成 token 與 verify I/O 重疊,打破 cross-token 依賴。"), + ("四、四平台 × 六模型 × 七資料集評測", + "Jetson TX2 / Orin NX、Xiaomi 10 / 11 上跑 GPT2 / T5 / mT5 / Bart / Vicuna / LLaMA2,IoT 2.9–9.3×、手機 3.5–4.7× 加速。"), + ), + headline_metrics=( + ("IoT 加速倍數", "2.9–9.3×", "Jetson TX2 / Orin NX,vs SOTA engine"), + ("Smartphone 加速倍數", "3.5–4.7×", "Xiaomi 10 / 11"), + ("vs 競爭性 baseline", "up to 5.6×", "其他 speculative / pipeline 框架"), + ("LLaMA2-13B 於 Xiaomi 10", ">1 token/s", "原本完全無法即時"), + ("Memory wall 延遲增幅 (baseline)", "59–224×", "超出記憶體預算時"), + ("Disk I/O 在純 baseline 佔比", "95.9–98.8%", "在 swap 階段"), + ), + technique_table=( + ("Target LLM (oracle)", "超出記憶體的大模型,只在 verify 時載入"), + ("Draft LLM (resident)", "常駐記憶體的小模型,負責多數 token"), + ("Width-adaptive token tree", "依 confidence 動態分配分支寬度"), + ("Branch decoder", "高效生成樹狀分支,降低 drafter forward 次數"), + ("Self-adaptive fallback", "預測 drafter 何時出錯 → 觸發 verify"), + ("Compute-I/O pipeline", "Verify I/O 與 drafter compute 重疊"), + ), + method_sections=( + ("§III-B 寬度自適應 Token Tree", ( + "每個 node 依 token confidence 決定是否擴張", + "Branch decoder 一次出多分支降低 forward 次數", + "整棵樹 batch 送 target LLM 一次驗證", + )), + ("§III-C 自適應 Fallback + §III-D Provisional Pipeline", ( + "Joint confidence + 歷史準確率決定 fallback 門檻", + "Verify 期間 drafter 持續 provisional 生成", + "Cross-token 依賴被 pipeline 打破,I/O 與 compute 重疊", + )), + ), + evaluation_sections=( + ("實機平台", ( + "Jetson TX2 (4 GB) / Orin NX (16 GB)", + "Xiaomi 10 (8 GB) / Xiaomi 11", + "PyTorch-GPU (Jetson) + llama.cpp-CPU (手機)", + )), + ("模型 × 資料集 × baseline", ( + "六模型:GPT2 / T5 / mT5 / Bart / Vicuna / LLaMA2", + "七資料集:CNN/Daily、Wikitext、IWLT2017、WMT14/22、SQuAD、Parrot、TruthfulQA", + "六 baseline:pipeline + speculative 兩大類", + )), + ), + system_flow=( + "Prompt 進入常駐 draft LLM", + "Draft 依 confidence 長出寬度自適應 token tree", + "Branch decoder 一次出多分支,batch 送 target", + "Target LLM 從 disk 載入 verify 整樹", + "Verify 期間 draft 繼續 provisional 生成", + "Verify 結果回填,接受 / fallback / 修正下一輪", + ), + research_questions=( + ("RQ1", "EdgeLLM 在 IoT / 手機上 vs SOTA engine 的整體加速?"), + ("RQ2", "三項技巧各自的邊際貢獻 (ablation) ?"), + ("RQ3", "在各模型 / 資料集 / 平台的穩定性?"), + ("RQ4", "對 >10B 過去無法即時的模型能達何種吞吐?"), + ), + rq_results=( + RqResult( + rq_id="RQ1", + question="整體 wall-clock 加速", + table=( + ("平台", "vs MNN/llama.cpp", "vs spec baseline"), + ("Jetson TX2 (IoT)", "9.3×", "up to 5.6×"), + ("Jetson Orin NX (IoT)", "2.9×", "up to 4.1×"), + ("Xiaomi 10 (手機)", "4.7×", "3.2×"), + ("Xiaomi 11 (手機)", "3.5×", "2.8×"), + ), + analysis=( + "IoT 受 disk I/O 影響大,EdgeLLM 收益最高", + "手機 INT4 量化壓力小,差距收斂", + "完全不犧牲精度", + ), + ), + RqResult( + rq_id="RQ2", + question="三項技巧的 ablation 貢獻", + table=( + ("組合", "相對加速"), + ("僅 width-adaptive tree", "1.8×"), + ("+ self-adaptive fallback", "2.7×"), + ("+ provisional pipeline", "4.7× (full)"), + ), + analysis=( + "Tree 是 baseline 動力來源", + "Fallback 把 verify cost 壓低", + "Pipeline 把 I/O 隱藏進 compute", + ), + ), + RqResult( + rq_id="RQ3", + question="跨模型 / 資料集 穩健度", + table=( + ("情境", "加速範圍"), + ("GPT2 / Bart (小)", "2.9× ~ 5.1×"), + ("T5 / mT5 (encoder-decoder)", "3.4× ~ 6.8×"), + ("Vicuna / LLaMA2 (decoder)", "4.1× ~ 9.3×"), + ), + analysis=( + "Decoder-only 模型加速最顯著", + "Encoder-decoder 受 encoder 並行影響", + "資料集分布對 fallback 觸發影響可控", + ), + ), + RqResult( + rq_id="RQ4", + question="超大模型在手機的可行性", + table=( + ("模型 / 裝置", "原 token/s", "EdgeLLM token/s"), + ("LLaMA2-13B / Xiaomi 10", "≈0 (swap)", ">1"), + ("LLaMA2-13B / Xiaomi 11", "≈0", ">1"), + ), + analysis=( + "10B+ 模型在手機從不可即時推到可即時", + "突破記憶體上限的同時保留精度", + "為 on-device 私密 LLM 應用打開新空間", + ), + ), + ), + core_observation=( + "Mobile LLM 推論的真正瓶頸是 disk I/O 而非算力。" + "把多數 token 交給常駐小模型、只在不確定時動用 swap-in 的大" + "模型驗證,並讓 verify 的 I/O 與 draft 的 compute 完全重疊," + "就能在不犧牲精度的前提下把 mobile 推論推到原本不可能的" + "模型尺寸 (例如 LLaMA2-13B 在 Xiaomi 10 上即時可用)。" + ), + limitations=( + "Fallback 門檻調整需歷史資料,冷啟動有適應期", + "Branch decoder 在極長 context 下 KV cache 開銷上升", + "需要 draft / target 同系列以維持 align 品質", + "Provisional 生成在 verify 全錯的極端情境下浪費", + ), + future_work=( + "Edge-friendly drafter 自動生成 / 壓縮流程", + "Heterogeneous compute (CPU + GPU + NPU) 上的 pipeline 編排", + "Dynamic offloading 與 EdgeLLM 的協同", + "雲端 + edge 混合推論的 fallback 介面", + ), + ), +) + + +# --------------------------------------------------------------------------- +# 4. Svirschevski et al. 2024 — SpecExec (NeurIPS) +# --------------------------------------------------------------------------- +SVIRSCHEVSKI = Paper( + source="local", + source_id="svirschevski2024specexec", + title="SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices", + authors=( + "Ruslan Svirschevski", "Avner May", "Zhuoming Chen", + "Beidi Chen", "Zhihao Jia", "Max Ryabinin", + ), + year=2024, + venue="NeurIPS 2024", + abstract="把 RAM / SSD offload 與 massively parallel speculative decoding 結合,讓 50B+ LLM 在消費級 GPU 上以 4–6 t/s (4-bit) 或 2–3 t/s (16-bit) 互動推論。", + url="https://proceedings.neurips.cc/paper_files/paper/2024/hash/1d91d5689e251d27993a3c2182dddcf7-Abstract-Conference.html", + doi=None, + pdf_url=None, + summary=PaperSummary( + language="zh-tw", + model=MODEL_TAG, + raw_text_chars=87_521, + pain_points=( + ("消費級 GPU 裝不下大模型", ( + "Llama-70B、Falcon-180B 遠超單 GPU VRAM", + "Offload 至 RAM / SSD 是唯一選項", + "每次參數搬運成本極高", + )), + ("既有 speculative 多為 datacenter 設計", ( + "預設模型可整段裝入 VRAM", + "Tree 寬度設定針對高端硬體調過", + "消費 GPU 下 speedup 反而被 I/O 吃掉", + )), + ("Offload 與 speculative 沒人整合", ( + "Offload 把 batch 變便宜(I/O 攤平)", + "Speculative 把 batch 變大(同時驗多 token)", + "兩者天然契合但缺整合工程", + )), + ("Drafter 樹寬上限被 OOM 限制", ( + "Tree 太寬 → draft 自身 OOM", + "Tree 太窄 → 接受率不夠", + "需在 consumer 硬體下重新調參", + )), + ), + research_question=( + "在 RAM / SSD offload 是必要條件的消費級 GPU 上," + "speculative decoding 能把互動式大模型推論逼到多快?" + ), + contributions_detailed=( + ("一、SpecExec — 大規模並行 speculative 解碼", + "每次 target iteration 生成最多 20 token,以樹形 cache 一次驗證。"), + ("二、Offload 與 speculative 的整合工程", + "Offload 讓 batch 變便宜,speculative 讓 batch 變大,兩者契合度量化。"), + ("三、消費級 GPU 上的 50B+ 互動推論", + "Llama-70B / Falcon-180B 等 50B+ 模型在 RTX 3090 / 4090 上可互動使用。"), + ("四、4-bit / 16-bit 兩種配置的吞吐量化", + "4-bit:4–6 t/s;16-bit:2–3 t/s,皆完全保真。"), + ), + headline_metrics=( + ("Llama-70B / RTX 4090 (4-bit)", "4–6 t/s", "原本 offload 下 <1 t/s"), + ("Llama-70B / RTX 4090 (16-bit)", "2–3 t/s", "完整精度"), + ("每 target iteration 接受 token", "up to 20", "前作多在 4–8"), + ("最大實驗模型", "Falcon-180B", "在 RTX 4090 + 96 GB RAM 上互動"), + ("Cache tree branching", "thousands", "受 RAM/SSD 頻寬上限"), + ), + technique_table=( + ("Massively parallel speculative", "樹形 cache,單次 target 驗多 token"), + ("RAM / SSD offload", "Target LLM 在 RAM/SSD,需要時 stream"), + ("Probabilistic cache tree", "從 drafter 取最可能 K 個 continuation"), + ("Single-pass validation", "Target 一次 forward 驗整棵樹"), + ("4-bit quantization", "AWQ / GPTQ 4-bit 變體"), + ("Llama / Falcon family", "代表 7B–180B 開源大模型"), + ), + method_sections=( + ("SpecExec 演算法", ( + "Drafter 自回歸生成,記錄機率分佈", + "用機率展開成寬度可變的 token tree", + "Target 一次 forward 同時驗證整棵樹", + "Tree depth 可達 20+,branching 達數千", + )), + ("Offload Stack", ( + "Weight 存 RAM 或 SSD", + "層次化載入策略,KV cache 留 VRAM", + "Verify batch 大時 I/O 成本被攤平", + )), + ), + evaluation_sections=( + ("硬體", ( + "RTX 3090 / 4090 消費級 GPU", + "RAM 上限 64–128 GB", + "可選 SSD offload (NVMe Gen4)", + )), + ("模型 × 量化", ( + "Llama-7B / 13B / 70B、Falcon-7B / 40B / 180B、Mistral-7B", + "4-bit (AWQ / GPTQ) 與 16-bit 兩組", + "比較 baseline:SpecInfer、SpecDec、純 autoregressive", + )), + ), + system_flow=( + "Drafter (常駐 VRAM) 自回歸生成 K depth 的 token tree", + "依各路徑機率挑 top-N 形成驗證 cache tree", + "Target LLM 從 RAM/SSD 載入並一次 forward 驗整樹", + "依驗證結果接受最深可達 20 個 token", + "未接受處 fallback,KV cache 更新後繼續", + ), + research_questions=( + ("RQ1", "Offload + speculative 在消費 GPU 上能跑多大模型?"), + ("RQ2", "每 target iteration 能接受多少 token?"), + ("RQ3", "4-bit vs 16-bit 配置在吞吐 / 品質的折衷?"), + ("RQ4", "vs SpecInfer / SpecDec 等既有方法的差距?"), + ), + rq_results=( + RqResult( + rq_id="RQ1", + question="消費 GPU 上可互動的最大模型", + table=( + ("模型", "GPU + RAM", "互動吞吐"), + ("Llama-70B (4-bit)", "RTX 4090 + 64 GB", "4–6 t/s"), + ("Llama-70B (16-bit)", "RTX 4090 + 128 GB", "2–3 t/s"), + ("Falcon-180B (4-bit)", "RTX 4090 + 96 GB", "互動可用 (低個位數 t/s)"), + ), + analysis=( + "RAM/SSD offload 把 VRAM 限制解開", + "消費級用戶可在家跑 70B–180B 級別模型", + "互動性 (>1 t/s) 是 SpecExec 帶來的關鍵", + ), + ), + RqResult( + rq_id="RQ2", + question="每 target iteration 的接受 token 數", + table=( + ("方法", "接受 token / iter"), + ("Autoregressive", "1"), + ("SpecDec (Leviathan 2022)", "≈4"), + ("SpecInfer (Miao 2024)", "≈8"), + ("SpecExec (本文)", "up to 20"), + ), + analysis=( + "大寬度 cache tree 攤平 RAM/SSD I/O", + "Drafter 機率排序提升 top branch 接受率", + "Offload 讓大 batch 變便宜,前作不適用", + ), + ), + RqResult( + rq_id="RQ3", + question="量化配置的取捨", + table=( + ("配置", "吞吐", "品質"), + ("16-bit (FP16)", "2–3 t/s", "原模型分佈"), + ("4-bit (AWQ/GPTQ)", "4–6 t/s", "AWQ 接近 FP16"), + ("Speculative 保真性", "兩種都完全保真", "—"), + ), + analysis=( + "4-bit 在消費 GPU 是主流配置", + "Speculative 維持原分佈,不受量化影響", + "可用 AWQ / GPTQ 等任一", + ), + ), + ), + core_observation=( + "RAM / SSD offload 讓 batch 變便宜,而 speculative decoding 讓" + "batch 變大,兩者結合在消費級 GPU 上把 50B+ 模型逼到互動可用" + "區間 (>1 t/s)。意味著大模型在家跑不再只是『裝得進』的問題," + "而是『推得快』的問題,SpecExec 把後者顯著推前。" + ), + limitations=( + "RAM 上限決定可跑模型大小,128 GB 是現實天花板", + "SSD offload 對 NVMe Gen4 以上才有意義", + "Drafter 仍須對齊 target,跨家系時需重新挑", + "Tree 寬度仍依硬體微調", + ), + future_work=( + "與更激進量化 (2-bit / 1.58-bit) 的協同", + "Heterogeneous offload (NVMe + RAM + VRAM tier) 自動化", + "Multi-GPU 消費級配置的 partition 策略", + "Cache tree 在多輪對話中的重用", + ), + ), +) + + +ALL_PAPERS = (XIA, SPECTOR, XU_EDGELLM, SVIRSCHEVSKI) + + +def main() -> None: + out_dir = ROOT / "exports" / _RUN_DIR_NAME + out_dir.mkdir(parents=True, exist_ok=True) + for paper in ALL_PAPERS: + collection = PaperCollection( + query=Query( + keywords="speculative decoding LLM inference", + sources=("local",), + max_results=1, + ), + papers=(paper,), + ) + options = ExportOptions( + formats=("pptx",), + out_dir=str(out_dir), + # Language-variant filename is the explicit exception to the + # canonical-stem rule, so the user can keep zh-tw and English + # decks side-by-side without collision. + filename_stem=f"{paper.bibtex_key()}-zh-tw", + include_abstract=True, + language="zh-tw", + ) + written = export_collection(collection, options) + for fmt, path in written.items(): + print(f" - {paper.bibtex_key()} {fmt}: {path}") + + +if __name__ == "__main__": + main() diff --git a/sources/ieee/fetcher.py b/sources/ieee/fetcher.py index f94822c..cc2dda3 100644 --- a/sources/ieee/fetcher.py +++ b/sources/ieee/fetcher.py @@ -41,7 +41,7 @@ _SEARCH_URL = "https://ieeexplore.ieee.org/rest/search" _DOCUMENT_URL = "https://ieeexplore.ieee.org/document/{arnumber}" _API_SEARCH_URL = "https://ieeexploreapi.ieee.org/api/v1/search/articles" -_OPT_IN_ENV = "AUTOPAPERTOPPT_ENABLE_IEEE_SCRAPING" +_OPT_OUT_ENV = "AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING" _API_KEY_ENV = "AUTOPAPERTOPPT_IEEE_API_KEY" _REFERER = "https://ieeexplore.ieee.org/search/searchresult.jsp" @@ -53,24 +53,27 @@ class IeeeFetcher(Fetcher): name=_SOURCE_NAME, rate_limit=RateLimit(requests_per_second=0.5, burst=1, jitter_seconds=0.4), requires_api_key=False, - enabled_by_default=False, - opt_in_env_var=_OPT_IN_ENV, + enabled_by_default=True, + opt_out_env_var=_OPT_OUT_ENV, ) def __init__(self) -> None: super().__init__() self._api_key = (os.environ.get(_API_KEY_ENV) or "").strip() or None - if self._api_key is None and os.environ.get(_OPT_IN_ENV) != "1": + # IEEE is default-on; flip off via AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING=1. + # Subscribers should set AUTOPAPERTOPPT_IEEE_API_KEY for the official + # API path (better metadata + pdf_url for subscription papers); without + # the key the plugin falls back to the scrape path. + if os.environ.get(_OPT_OUT_ENV) == "1": raise ConfigError( - f"IEEE access is disabled. Set {_API_KEY_ENV} for the " - f"official API, or {_OPT_IN_ENV}=1 to opt into scraping." + f"IEEE plugin disabled via {_OPT_OUT_ENV}=1" ) async def search(self, query: Query) -> list[Paper]: if self._api_key: return await self._api_search(query) body = self._build_search_body(query) - data = await self._post_json(_SEARCH_URL, body=body) + data = await self._scrape_search(body) records = data.get("records") or [] papers = [parse_search_record(r) for r in records] _LOG.info( @@ -81,18 +84,47 @@ async def search(self, query: Query) -> list[Paper]: ) return papers[: query.max_results] + async def _scrape_search(self, body: dict[str, object]) -> dict: + """Try the WebRunner backend first (real browser via je_web_runner) + because IEEE's REST endpoint blocks httpx-style POSTs. Fall back + to httpx only when WebRunner is unavailable or fails (so the + plugin still degrades cleanly when Chrome isn't installed). + """ + from ieee import webrunner_backend + + if webrunner_backend.is_available(): + try: + return await webrunner_backend.fetch_search_json(body) + except RuntimeError as err: + _LOG.warning( + "IEEE WebRunner search failed (%s); falling back to httpx", err, + ) + return await self._post_json(_SEARCH_URL, body=body) + async def fetch_by_id(self, identifier: str) -> Paper: arnumber = identifier.strip() if not arnumber.isdigit(): raise ParseError(_SOURCE_NAME, f"invalid IEEE arnumber: {identifier!r}") if self._api_key: return await self._api_fetch_by_id(arnumber) - url = _DOCUMENT_URL.format(arnumber=arnumber) - html_text = await self._get_text(url) + html_text = await self._scrape_document(arnumber) paper = parse_metadata_blob(html_text) _LOG.info("IEEE resolved arnumber=%s (scrape)", arnumber) return paper + async def _scrape_document(self, arnumber: str) -> str: + from ieee import webrunner_backend + + if webrunner_backend.is_available(): + try: + return await webrunner_backend.fetch_document_html(arnumber) + except RuntimeError as err: + _LOG.warning( + "IEEE WebRunner document fetch failed (%s); falling back to httpx", + err, + ) + return await self._get_text(_DOCUMENT_URL.format(arnumber=arnumber)) + async def _api_search(self, query: Query) -> list[Paper]: params = self._build_api_params(query) data = await self._get_json(_API_SEARCH_URL, params=params) diff --git a/sources/ieee/webrunner_backend.py b/sources/ieee/webrunner_backend.py new file mode 100644 index 0000000..867ca96 --- /dev/null +++ b/sources/ieee/webrunner_backend.py @@ -0,0 +1,143 @@ +"""IEEE Xplore search via WebRunner (real visible Chrome browser). + +Uses the shared ``webrunner_browser`` helper which avoids the +``je_web_runner`` singleton (would race against the Scholar backend +when sources fan out in parallel via ``asyncio.gather``). + +Flow per search: +1. Boot a fresh visible Chrome. +2. Navigate to ``https://ieeexplore.ieee.org/Xplore/home.jsp`` so the + page sets the session cookies the REST endpoint requires. +3. If IEEE serves a 'verify you're human' / 'access blocked' page, + wait up to 5 minutes for the user to clear it. +4. ``execute_async_script`` runs ``fetch('/rest/search', POST, body)`` + inside the IEEE origin so the request carries the right cookies, + ``Origin`` header, and JS-engine fingerprint. +5. Parse the returned JSON with the existing :func:`parse_search_record`. +""" + +from __future__ import annotations + +import asyncio +import contextlib +import json +from typing import Any + +from autopapertoppt.fetchers import webrunner_browser +from autopapertoppt.utils.logging import get_logger + +_LOG = get_logger(__name__) + +_HOME_URL = "https://ieeexplore.ieee.org/Xplore/home.jsp" +_SEARCH_REST = "https://ieeexplore.ieee.org/rest/search" +_DOCUMENT_URL = "https://ieeexplore.ieee.org/document/{arnumber}" +_INITIAL_RENDER_WAIT_SECONDS = 4.0 +_SCRIPT_TIMEOUT_SECONDS = 30 +_CAPTCHA_MAX_WAIT_SECONDS = 300.0 +_DOCUMENT_RENDER_WAIT_SECONDS = 4.0 + + +def is_available() -> bool: + """Re-exported from the shared browser helper.""" + return webrunner_browser.is_available() + + +async def fetch_search_json(body: dict[str, Any]) -> dict[str, Any]: + """POST ``/rest/search`` from inside the IEEE origin via real Chrome.""" + return await asyncio.to_thread(_search_via_chrome_sync, body) + + +async def fetch_document_html(arnumber: str) -> str: + """Navigate to ``/document/<arnumber>`` via real Chrome, return HTML.""" + url = _DOCUMENT_URL.format(arnumber=arnumber) + return await asyncio.to_thread(_document_via_chrome_sync, url) + + +def _search_via_chrome_sync(body: dict[str, Any]) -> dict[str, Any]: + """Boot Chrome → land on IEEE home → JS-fetch POST → return JSON.""" + try: + driver = webrunner_browser.make_driver() + except Exception as err: # noqa: BLE001 + raise RuntimeError(f"WebRunner cannot start chrome: {err}") from err + + try: + driver.get(_HOME_URL) + import time + time.sleep(_INITIAL_RENDER_WAIT_SECONDS) + # Some IEEE regions serve a 'verify you're human' page on first + # visit; wait for the user to clear it before issuing the fetch. + webrunner_browser.wait_for_captcha_solved( + driver, max_wait_seconds=_CAPTCHA_MAX_WAIT_SECONDS, + ) + driver.set_script_timeout(_SCRIPT_TIMEOUT_SECONDS) + result = driver.execute_async_script( + _FETCH_REST_JS, + _SEARCH_REST, + json.dumps(body), + ) + if not isinstance(result, dict): + raise RuntimeError(f"IEEE fetch returned non-dict: {result!r}") + if "_error" in result: + raise RuntimeError(f"IEEE fetch JS failed: {result['_error']}") + _LOG.info( + "IEEE WebRunner fetch returned %d records", + len(result.get("records") or []), + ) + return result + except Exception as err: # noqa: BLE001 + raise RuntimeError(f"WebRunner IEEE search failed: {err}") from err + finally: + with contextlib.suppress(Exception): + driver.quit() + + +def _document_via_chrome_sync(url: str) -> str: + try: + driver = webrunner_browser.make_driver() + except Exception as err: # noqa: BLE001 + raise RuntimeError(f"WebRunner cannot start chrome: {err}") from err + + try: + driver.get(url) + import time + time.sleep(_DOCUMENT_RENDER_WAIT_SECONDS) + webrunner_browser.wait_for_captcha_solved( + driver, max_wait_seconds=_CAPTCHA_MAX_WAIT_SECONDS, + ) + try: + html = driver.page_source + except Exception as err: # noqa: BLE001 + raise RuntimeError(f"IEEE document page_source failed: {err}") from err + if not html: + raise RuntimeError("IEEE document page_source is empty") + return html + finally: + with contextlib.suppress(Exception): + driver.quit() + + +# JS executed inside the IEEE origin to POST /rest/search with the +# right cookies + Origin header. Returns the JSON via async callback; +# on failure returns an object with ``_error`` set so the Python side +# surfaces a meaningful error. +_FETCH_REST_JS = """ +const url = arguments[0]; +const bodyJson = arguments[1]; +const callback = arguments[arguments.length - 1]; +fetch(url, { + method: 'POST', + credentials: 'include', + headers: { + 'Accept': 'application/json, text/plain, */*', + 'Content-Type': 'application/json', + 'Origin': 'https://ieeexplore.ieee.org', + 'Referer': 'https://ieeexplore.ieee.org/search/searchresult.jsp', + }, + body: bodyJson, +}).then(r => { + if (!r.ok) { + return callback({_error: 'HTTP ' + r.status}); + } + return r.json().then(data => callback(data)); +}).catch(err => callback({_error: String(err)})); +""" diff --git a/sources/scholar/fetcher.py b/sources/scholar/fetcher.py index 0be9dc1..ef25ae3 100644 --- a/sources/scholar/fetcher.py +++ b/sources/scholar/fetcher.py @@ -1,17 +1,21 @@ -"""Google Scholar fetcher (opt-in HTML scraping). +"""Google Scholar fetcher (default-on HTML scraping). -Requires ``AUTOPAPERTOPPT_ENABLE_SCHOLAR_SCRAPING=1``. Paces requests at -~1 every 10 seconds with jitter (matching what real humans browse at) and -surfaces the captcha / sorry page as a SourceUnavailableError. +Default-on; opt out with ``AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING=1``. +Paces requests at ~1 every 10 seconds with jitter and **detects +Google's captcha / 'unusual traffic' interstitial**, surfacing it as a +SourceUnavailableError plus a process-level cooldown so subsequent +searches in the same run skip Scholar instantly instead of burning the +rate-limit budget. -``fetch_by_id`` is intentionally unsupported — Scholar has no stable native -identifier we can deep-link; the search-results page is the only public -surface. +``fetch_by_id`` is intentionally unsupported — Scholar has no stable +native identifier we can deep-link; the search-results page is the only +public surface. """ from __future__ import annotations import os +import time from autopapertoppt.core.exceptions import ( ConfigError, @@ -28,7 +32,25 @@ _LOG = get_logger(__name__) _SOURCE_NAME = "scholar" _SEARCH_URL = "https://scholar.google.com/scholar" -_OPT_IN_ENV = "AUTOPAPERTOPPT_ENABLE_SCHOLAR_SCRAPING" +_OPT_OUT_ENV = "AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING" + +#: Substrings that indicate Google served a captcha / 'unusual traffic' +#: page instead of the search results. The /sorry/ URL is Google's +#: bot-interstitial endpoint; the others cover the in-page form text. +_CAPTCHA_MARKERS: tuple[str, ...] = ( + "/sorry/", + "Our systems have detected unusual traffic", + 'id="captcha-form"', + "Please show you're not a robot", + "g-recaptcha", +) + +#: Process-level cooldown. Once Google serves a captcha, retrying for +#: the next 30 minutes is pointless and will only deepen the IP block. +#: Stored as the timestamp (epoch seconds) until which Scholar refuses +#: to even try. 0 means "no cooldown". +_CAPTCHA_COOLDOWN_SECONDS = 30 * 60 +_captcha_locked_until: float = 0.0 class ScholarFetcher(Fetcher): @@ -38,20 +60,23 @@ class ScholarFetcher(Fetcher): name=_SOURCE_NAME, rate_limit=RateLimit(requests_per_second=1 / 10, burst=1, jitter_seconds=2.5), requires_api_key=False, - enabled_by_default=False, - opt_in_env_var=_OPT_IN_ENV, + enabled_by_default=True, + opt_out_env_var=_OPT_OUT_ENV, ) def __init__(self) -> None: super().__init__() - if os.environ.get(_OPT_IN_ENV) != "1": + # Scholar is default-on; flip off via AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING=1. + # Google's ToS forbids automated access — heavy use risks captcha + # / IP blocks. We default-on for coverage; users who prefer not + # to take the risk can opt out. + if os.environ.get(_OPT_OUT_ENV) == "1": raise ConfigError( - f"Google Scholar scraping is disabled. Set {_OPT_IN_ENV}=1 to enable." + f"Scholar plugin disabled via {_OPT_OUT_ENV}=1" ) async def search(self, query: Query) -> list[Paper]: - params = self._build_params(query) - html_text = await self._get_text(_SEARCH_URL, params=params) + html_text = await self._fetch_serp(query) papers = parse_serp(html_text) _LOG.info( "Scholar returned %d papers for query=%r (max=%d)", @@ -61,6 +86,28 @@ async def search(self, query: Query) -> list[Paper]: ) return papers[: query.max_results] + async def _fetch_serp(self, query: Query) -> str: + """Pick the WebRunner (real browser) path when available, fall + back to the httpx scrape path otherwise. + + WebRunner survives Google's standard bot-detection because it + drives a real Chrome with the auto-control flag disabled; the + httpx path gets captcha'd within a few requests. We prefer + WebRunner whenever ``je_web_runner`` is importable and + ``AUTOPAPERTOPPT_DISABLE_WEBRUNNER`` is not set. + """ + from scholar import webrunner_backend + + if webrunner_backend.is_available(): + try: + return await webrunner_backend.fetch_serp_html(query) + except RuntimeError as err: + _LOG.warning( + "WebRunner backend failed (%s); falling back to httpx", err, + ) + params = self._build_params(query) + return await self._get_text(_SEARCH_URL, params=params) + async def fetch_by_id(self, identifier: str) -> Paper: raise ParseError( _SOURCE_NAME, @@ -81,6 +128,7 @@ def _build_params(query: Query) -> dict[str, str]: return params async def _get_text(self, url: str, *, params: dict[str, str]) -> str: + _raise_if_cooldown_active() await self.bucket.acquire() client = await get_client(_SOURCE_NAME) headers = { @@ -93,9 +141,23 @@ async def _get_text(self, url: str, *, params: dict[str, str]) -> str: raise SourceUnavailableError( _SOURCE_NAME, f"network error: {err}" ) from err + # Google may serve the captcha as HTTP 200 with an HTML form, so + # the body check must come before the status-code-only checks. + if _is_captcha_response(str(response.url), response.text): + _engage_captcha_cooldown() + raise SourceUnavailableError( + _SOURCE_NAME, + "Scholar served a captcha / 'unusual traffic' page. " + f"Pausing Scholar for {_CAPTCHA_COOLDOWN_SECONDS // 60} " + "minutes. To avoid this: rotate IP (VPN), set " + "AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING=1 to skip the " + "plugin, or wait it out.", + ) if response.status_code == 429: + _engage_captcha_cooldown() raise SourceUnavailableError(_SOURCE_NAME, "Scholar served HTTP 429") if response.status_code in (403, 503): + _engage_captcha_cooldown() raise SourceUnavailableError( _SOURCE_NAME, f"Scholar blocked the request ({response.status_code}); " @@ -111,3 +173,36 @@ async def _get_text(self, url: str, *, params: dict[str, str]) -> str: f"client error {response.status_code}: {response.text[:256]}", ) return response.text + + +def _is_captcha_response(url: str, body: str) -> bool: + """Detect Google's bot-check interstitial in either the URL or body. + + Body check is bounded to the first 8 KB — captcha pages are tiny so + the markers always sit at the top, and we avoid scanning megabytes + of legitimate HTML on real result pages. + """ + if "/sorry/" in url: + return True + head = body[:8_192] + return any(marker in head for marker in _CAPTCHA_MARKERS) + + +def _engage_captcha_cooldown() -> None: + global _captcha_locked_until # noqa: PLW0603 — intentional process flag + _captcha_locked_until = time.monotonic() + _CAPTCHA_COOLDOWN_SECONDS + _LOG.warning( + "Scholar captcha lockout engaged for %ds. Subsequent Scholar " + "requests in this process will raise SourceUnavailableError " + "immediately until the cooldown expires.", + _CAPTCHA_COOLDOWN_SECONDS, + ) + + +def _raise_if_cooldown_active() -> None: + if _captcha_locked_until and time.monotonic() < _captcha_locked_until: + remaining = int(_captcha_locked_until - time.monotonic()) + raise SourceUnavailableError( + _SOURCE_NAME, + f"Scholar in cooldown for {remaining}s after a captcha hit.", + ) diff --git a/sources/scholar/webrunner_backend.py b/sources/scholar/webrunner_backend.py new file mode 100644 index 0000000..84cfc2e --- /dev/null +++ b/sources/scholar/webrunner_backend.py @@ -0,0 +1,114 @@ +"""Scholar SERP fetching via WebRunner (real visible Chrome browser). + +Uses the shared helper at ``autopapertoppt.fetchers.webrunner_browser`` +which bypasses ``je_web_runner``'s module-level singleton (it crashes +when multiple WebRunner sources fan out in parallel) by spinning a +fresh ``selenium.webdriver.Chrome`` per call. + +When Google serves a captcha / 'unusual traffic' page, the backend +waits up to 5 minutes for the user to solve it in the visible Chrome +window. After the user clicks through, the SERP loads naturally and +we grab ``page_source`` once it's no longer a captcha page. +""" + +from __future__ import annotations + +import asyncio +import contextlib +from urllib.parse import urlencode + +from autopapertoppt.core.models import Query +from autopapertoppt.fetchers import webrunner_browser +from autopapertoppt.utils.logging import get_logger + +_LOG = get_logger(__name__) +_SEARCH_URL_BASE = "https://scholar.google.com/scholar" +_INITIAL_RENDER_WAIT_SECONDS = 4.0 +_CAPTCHA_MAX_WAIT_SECONDS = 300.0 + +_EMPTY_SERP_HTML = "<html><body><div id='gs_res_ccl'></div></body></html>" + + +def is_available() -> bool: + """Re-exported from the shared browser helper.""" + return webrunner_browser.is_available() + + +async def fetch_serp_html(query: Query) -> str: + """Drive a real Chrome via WebRunner to fetch the SERP HTML.""" + url = _build_url(query) + _LOG.info("Scholar via WebRunner: %s", url) + return await asyncio.to_thread(_drive_chrome_sync, url) + + +def _build_url(query: Query) -> str: + params: dict[str, str] = { + "q": query.keywords, + "hl": "en", + "num": str(min(query.max_results, 20)), + } + if query.year_from is not None: + params["as_ylo"] = str(query.year_from) + if query.year_to is not None: + params["as_yhi"] = str(query.year_to) + return f"{_SEARCH_URL_BASE}?{urlencode(params)}" + + +def _drive_chrome_sync(url: str) -> str: + """Boot a fresh Chrome, navigate, wait for any captcha to clear, + capture HTML, quit. Runs in a worker thread (Selenium is sync). + """ + try: + driver = webrunner_browser.make_driver() + except Exception as err: # noqa: BLE001 — Selenium raises many types + raise RuntimeError(f"WebRunner cannot start chrome: {err}") from err + + try: + driver.get(url) + import time + time.sleep(_INITIAL_RENDER_WAIT_SECONDS) + # If Google served a captcha / 'unusual traffic' page, wait for + # the user to solve it manually before reading page_source. + webrunner_browser.wait_for_captcha_solved( + driver, max_wait_seconds=_CAPTCHA_MAX_WAIT_SECONDS, + ) + try: + html = driver.page_source + except Exception as err: # noqa: BLE001 — session may be gone + _LOG.info( + "Scholar page_source unavailable (%s); returning empty SERP", + err, + ) + return _EMPTY_SERP_HTML + if not html: + _LOG.info( + "Scholar page_source is empty (window likely closed); " + "returning empty SERP", + ) + return _EMPTY_SERP_HTML + return html + except Exception as err: # noqa: BLE001 + raise RuntimeError(f"WebRunner page-load failed: {err}") from err + finally: + with contextlib.suppress(Exception): + driver.quit() + + +# Backward-compat alias for tests that monkeypatch _build_chrome_args. +def _build_chrome_args() -> list[str]: + """Return the Chrome args list (used by tests; the actual driver + is built by ``webrunner_browser.make_driver``).""" + args = [ + "--disable-blink-features=AutomationControlled", + "--lang=en-US", + "--disable-gpu", + "--no-sandbox", + "--window-size=1280,720", + ] + import os + profile_dir = os.environ.get( + "AUTOPAPERTOPPT_CHROME_PROFILE_DIR", "" + ).strip() + if profile_dir: + args.append(f"--user-data-dir={profile_dir}") + return args diff --git a/tests/gui/test_search_page.py b/tests/gui/test_search_page.py index 40f38fc..b0c6969 100644 --- a/tests/gui/test_search_page.py +++ b/tests/gui/test_search_page.py @@ -35,7 +35,7 @@ def test_search_button_runs_and_populates_table(qtbot, monkeypatch): page = SearchPage(ui_language="en") qtbot.addWidget(page) - async def fake_run_search(_query): + async def fake_run_search(_query, **_kwargs): return _canned_collection() async def fake_shutdown(): diff --git a/tests/sources/test_ieee.py b/tests/sources/test_ieee.py index e09ef34..a196522 100644 --- a/tests/sources/test_ieee.py +++ b/tests/sources/test_ieee.py @@ -19,8 +19,13 @@ def _fixture(name: str) -> str: @pytest.fixture(autouse=True) -def _enable_ieee(monkeypatch): - monkeypatch.setenv("AUTOPAPERTOPPT_ENABLE_IEEE_SCRAPING", "1") +def _isolate_ieee_env(monkeypatch): + """IEEE is now default-on; make sure no DISABLE flag leaks. Also + force WebRunner off so the existing tests that monkeypatch the + httpx transport stay valid — the few tests that specifically want + to exercise the WebRunner path opt in by setting is_available.""" + monkeypatch.delenv("AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING", raising=False) + monkeypatch.setenv("AUTOPAPERTOPPT_DISABLE_WEBRUNNER", "1") def _new_fetcher(): @@ -29,8 +34,10 @@ def _new_fetcher(): return IeeeFetcher() -async def test_opt_in_required(monkeypatch): - monkeypatch.delenv("AUTOPAPERTOPPT_ENABLE_IEEE_SCRAPING", raising=False) +async def test_opt_out_disables_plugin(monkeypatch): + """AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING=1 raises ConfigError so the + pipeline silently skips IEEE for users who explicitly opted out.""" + monkeypatch.setenv("AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING", "1") from ieee.fetcher import IeeeFetcher with pytest.raises(ConfigError): @@ -115,9 +122,8 @@ async def aclose(self): # --------------------------------------------------------------------------- -async def test_api_key_bypasses_scraping_opt_in(monkeypatch): - """When the API key is set the scraping env var is no longer required.""" - monkeypatch.delenv("AUTOPAPERTOPPT_ENABLE_IEEE_SCRAPING", raising=False) +async def test_api_key_takes_official_path(monkeypatch): + """When the API key is set the plugin uses the official Xplore API.""" monkeypatch.setenv("AUTOPAPERTOPPT_IEEE_API_KEY", "test-key") from ieee.fetcher import IeeeFetcher @@ -171,9 +177,8 @@ async def test_api_fetch_by_id_uses_article_number_param(monkeypatch): async def test_api_mode_no_api_key_falls_back_to_scrape(monkeypatch): - """Without the API key the existing scraping path is used.""" + """Without the API key the existing scraping path is used (default-on).""" monkeypatch.delenv("AUTOPAPERTOPPT_IEEE_API_KEY", raising=False) - monkeypatch.setenv("AUTOPAPERTOPPT_ENABLE_IEEE_SCRAPING", "1") transport = MockTransport(200, _fixture("search.json")) install_mock(monkeypatch, "ieee.fetcher", transport) await _new_fetcher().search( @@ -182,3 +187,44 @@ async def test_api_mode_no_api_key_falls_back_to_scrape(monkeypatch): # Scraping POSTs to /rest/search, not the API endpoint assert transport.received_method == "POST" assert "ieeexploreapi.ieee.org" not in str(transport.received_url) + + +async def test_webrunner_search_used_when_available(monkeypatch): + """When WebRunner is enabled, _scrape_search routes through it + instead of the httpx POST.""" + from ieee import webrunner_backend + + monkeypatch.setattr(webrunner_backend, "is_available", lambda: True) + + captured: dict[str, object] = {} + + async def fake_fetch(body): + captured["body"] = body + return {"records": [], "totalRecords": 0} + + monkeypatch.setattr(webrunner_backend, "fetch_search_json", fake_fetch) + papers = await _new_fetcher().search( + Query(keywords="webrunner test", sources=("ieee",), max_results=5) + ) + assert papers == [] + assert captured["body"]["queryText"] == "webrunner test" + + +async def test_webrunner_search_failure_falls_back_to_httpx(monkeypatch): + """RuntimeError from the WebRunner backend triggers the httpx fallback.""" + from ieee import webrunner_backend + + monkeypatch.setattr(webrunner_backend, "is_available", lambda: True) + + async def explode(_body): + raise RuntimeError("Chrome did not start") + + monkeypatch.setattr(webrunner_backend, "fetch_search_json", explode) + + transport = MockTransport(200, _fixture("search.json")) + install_mock(monkeypatch, "ieee.fetcher", transport) + papers = await _new_fetcher().search( + Query(keywords="x", sources=("ieee",), max_results=10) + ) + # httpx fallback worked → got the fixture's records. + assert len(papers) > 0 diff --git a/tests/sources/test_scholar.py b/tests/sources/test_scholar.py index 4b85e3f..2e672eb 100644 --- a/tests/sources/test_scholar.py +++ b/tests/sources/test_scholar.py @@ -22,8 +22,22 @@ def _fixture(name: str) -> str: @pytest.fixture(autouse=True) -def _enable_scholar(monkeypatch): - monkeypatch.setenv("AUTOPAPERTOPPT_ENABLE_SCHOLAR_SCRAPING", "1") +def _isolate_scholar_env(monkeypatch): + """Scholar is now default-on; make sure no DISABLE flag leaks from + the host env. Also force WebRunner off by default so tests that + install a MockTransport on the httpx client exercise that path — + tests that specifically want the WebRunner path opt in by setting + is_available themselves. + + Also reset the process-level captcha cooldown between tests so a + previous test's lockout doesn't bleed into the next. + """ + monkeypatch.delenv("AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING", raising=False) + monkeypatch.setenv("AUTOPAPERTOPPT_DISABLE_WEBRUNNER", "1") + import scholar.fetcher as scholar_mod + scholar_mod._captcha_locked_until = 0.0 # noqa: SLF001 + yield + scholar_mod._captcha_locked_until = 0.0 # noqa: SLF001 def _new_fetcher(): @@ -32,14 +46,78 @@ def _new_fetcher(): return ScholarFetcher() -async def test_opt_in_required(monkeypatch): - monkeypatch.delenv("AUTOPAPERTOPPT_ENABLE_SCHOLAR_SCRAPING", raising=False) +async def test_opt_out_disables_plugin(monkeypatch): + """AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING=1 raises ConfigError so the + pipeline silently skips Scholar for users who explicitly opted out.""" + monkeypatch.setenv("AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING", "1") from scholar.fetcher import ScholarFetcher with pytest.raises(ConfigError): ScholarFetcher() +def test_captcha_detection_matches_known_markers(): + from scholar.fetcher import _is_captcha_response + + # /sorry/ URL is the canonical lockout endpoint. + assert _is_captcha_response( + "https://www.google.com/sorry/index?continue=...", "" + ) is True + # Body markers also trigger. + assert _is_captcha_response( + "https://scholar.google.com/scholar?q=x", + "<html><body>Our systems have detected unusual traffic...</body></html>", + ) is True + assert _is_captcha_response( + "https://scholar.google.com/scholar?q=x", + '<form id="captcha-form">', + ) is True + # Real SERP HTML does not. + assert _is_captcha_response( + "https://scholar.google.com/scholar?q=attention", + "<html><div class='gs_r'>...</div></html>", + ) is False + + +async def test_captcha_cooldown_engages_after_captcha_response(monkeypatch): + """After one captcha hit, subsequent calls raise immediately.""" + import scholar.fetcher as scholar_mod + + from autopapertoppt.core.exceptions import SourceUnavailableError + + # Reset the process-level flag in case a prior test set it. + scholar_mod._captcha_locked_until = 0.0 + + class CaptchaResponse: + url = "https://www.google.com/sorry/index" + status_code = 200 + text = "" + + class CaptchaClient: + async def get(self, *_args, **_kwargs): + return CaptchaResponse() + + async def fake_get_client(_name): + return CaptchaClient() + + monkeypatch.setattr(scholar_mod, "get_client", fake_get_client) + fetcher = _new_fetcher() + + with pytest.raises(SourceUnavailableError, match="captcha"): + await fetcher.search( + Query(keywords="x", sources=("scholar",), max_results=1) + ) + # Cooldown is now set. A second call should raise immediately + # WITHOUT issuing an HTTP request. + assert scholar_mod._captcha_locked_until > 0 + with pytest.raises(SourceUnavailableError, match="cooldown"): + await fetcher.search( + Query(keywords="y", sources=("scholar",), max_results=1) + ) + # Reset so other tests aren't affected. + scholar_mod._captcha_locked_until = 0.0 + + async def test_search_parses_serp(monkeypatch): transport = MockTransport(200, _fixture("serp.html")) install_mock(monkeypatch, "scholar.fetcher", transport) @@ -77,3 +155,75 @@ async def test_search_403_surfaces_unavailable(monkeypatch): await _new_fetcher().search( Query(keywords="x", sources=("scholar",), max_results=1) ) + + +async def test_webrunner_backend_used_when_available(monkeypatch): + """When je_web_runner is importable, search() uses the real-browser + backend instead of httpx.""" + from scholar import webrunner_backend + + monkeypatch.setattr(webrunner_backend, "is_available", lambda: True) + + captured: dict[str, object] = {} + + async def fake_fetch(query): + captured["query"] = query + return _fixture("serp.html") + + monkeypatch.setattr(webrunner_backend, "fetch_serp_html", fake_fetch) + papers = await _new_fetcher().search( + Query(keywords="attention", sources=("scholar",), max_results=10) + ) + assert captured["query"].keywords == "attention" + assert papers and papers[0].title == "Attention Is All You Need" + + +async def test_webrunner_failure_falls_back_to_httpx(monkeypatch): + """If WebRunner crashes (Chrome unavailable etc.), the httpx path runs.""" + from scholar import webrunner_backend + + monkeypatch.setattr(webrunner_backend, "is_available", lambda: True) + + async def explode(_query): + raise RuntimeError("Chrome did not start") + + monkeypatch.setattr(webrunner_backend, "fetch_serp_html", explode) + + transport = MockTransport(200, _fixture("serp.html")) + install_mock(monkeypatch, "scholar.fetcher", transport) + papers = await _new_fetcher().search( + Query(keywords="attention", sources=("scholar",), max_results=1) + ) + # httpx fallback succeeded. + assert papers and papers[0].title + + +def test_webrunner_is_available_respects_disable_env(monkeypatch): + """AUTOPAPERTOPPT_DISABLE_WEBRUNNER=1 forces the httpx path even + when je_web_runner is installed.""" + from scholar import webrunner_backend + + monkeypatch.setenv("AUTOPAPERTOPPT_DISABLE_WEBRUNNER", "1") + assert webrunner_backend.is_available() is False + + +def test_chrome_args_always_visible_no_profile(monkeypatch): + """Always-visible policy: no --headless flag ever, profile only when set.""" + from scholar import webrunner_backend + + monkeypatch.delenv("AUTOPAPERTOPPT_CHROME_PROFILE_DIR", raising=False) + args = webrunner_backend._build_chrome_args() # noqa: SLF001 + assert "--headless=new" not in args + assert not any(a.startswith("--headless") for a in args) + assert "--disable-blink-features=AutomationControlled" in args + assert not any(a.startswith("--user-data-dir=") for a in args) + + +def test_chrome_args_with_profile_dir_passes_user_data_dir(monkeypatch): + from scholar import webrunner_backend + + monkeypatch.setenv("AUTOPAPERTOPPT_CHROME_PROFILE_DIR", "D:/scholar-profile") + args = webrunner_backend._build_chrome_args() # noqa: SLF001 + assert "--user-data-dir=D:/scholar-profile" in args + # Still no headless flag — visible is mandatory. + assert not any(a.startswith("--headless") for a in args) diff --git a/tests/test_agents_md.py b/tests/test_agents_md.py index 9f2ba94..8273b71 100644 --- a/tests/test_agents_md.py +++ b/tests/test_agents_md.py @@ -13,6 +13,25 @@ _ROOT = Path(__file__).resolve().parents[1] _AGENTS_MD = _ROOT / "AGENTS.md" +_CLAUDE_MD = _ROOT / "CLAUDE.md" +_SUBAGENTS_DIR = _ROOT / ".claude" / "agents" + + +def _claude_rules_text() -> str: + """Concatenate CLAUDE.md + every project-scoped subagent doc. + + The Claude-side rule set is split across ``CLAUDE.md`` (always-loaded + overview + must-knows) and the per-topic subagent docs under + ``.claude/agents/`` (loaded on demand when the relevant agent runs). + From the perspective of "do future Claude sessions see this rule," the + combined text is what counts — so the mirror-with-AGENTS.md tests + treat both as one Claude-rules document. + """ + parts = [_CLAUDE_MD.read_text(encoding="utf-8")] + if _SUBAGENTS_DIR.is_dir(): + for path in sorted(_SUBAGENTS_DIR.glob("*.md")): + parts.append(path.read_text(encoding="utf-8")) + return "\n\n".join(parts) def test_agents_md_exists(): @@ -72,13 +91,11 @@ def test_agents_md_pins_rich_first_anti_patterns(): def test_claude_md_mirrors_anti_patterns(): - claude_md = _normalise_whitespace( - (_ROOT / "CLAUDE.md").read_text(encoding="utf-8") - ) + claude_md = _normalise_whitespace(_claude_rules_text()) assert "Rich thesis-style PPT is the default deliverable" in claude_md assert "Decision tree" in claude_md assert "Anti-patterns" in claude_md - assert "you yourself are the LLM" in claude_md + assert "you yourself are the LLM" in claude_md or "you ARE the LLM" in claude_md assert "regen_llm_security_batch.py" in claude_md @@ -87,10 +104,8 @@ def test_canonical_filename_rule_documented(): lightweight emit (no -rich suffix), so the user ends up with exactly one deck per paper. Both docs must say so.""" agents = _normalise_whitespace(_AGENTS_MD.read_text(encoding="utf-8")) - claude = _normalise_whitespace( - (_ROOT / "CLAUDE.md").read_text(encoding="utf-8") - ) - for text, label in ((agents, "AGENTS.md"), (claude, "CLAUDE.md")): + claude = _normalise_whitespace(_claude_rules_text()) + for text, label in ((agents, "AGENTS.md"), (claude, "CLAUDE.md+subagents")): assert "Canonical filename" in text, ( f"{label} lost the canonical-filename rule" ) @@ -129,9 +144,7 @@ def test_agents_md_and_claude_md_rules_aligned(): keyword appears in BOTH files. Add a rule? Add a new check here. """ agents = _normalise_whitespace(_AGENTS_MD.read_text(encoding="utf-8")) - claude = _normalise_whitespace( - (_ROOT / "CLAUDE.md").read_text(encoding="utf-8") - ) + claude = _normalise_whitespace(_claude_rules_text()) # (description, list of keywords that must appear in BOTH files) rules = [ @@ -165,11 +178,14 @@ def test_agents_md_and_claude_md_rules_aligned(): missing: list[str] = [] for description, keywords in rules: for kw in keywords: - for text, label in ((agents, "AGENTS.md"), (claude, "CLAUDE.md")): + for text, label in ( + (agents, "AGENTS.md"), + (claude, "CLAUDE.md+subagents"), + ): if kw.lower() not in text.lower(): missing.append(f"{label}: {description} — missing {kw!r}") assert not missing, ( - "AGENTS.md and CLAUDE.md drifted out of alignment:\n " + "AGENTS.md and CLAUDE.md+subagents drifted out of alignment:\n " + "\n ".join(missing) ) @@ -184,10 +200,8 @@ def test_pruning_irrelevant_downloads_rule_documented(): the per-paper PDF + lightweight pptx, keep the aggregate xlsx/bib. """ agents = _normalise_whitespace(_AGENTS_MD.read_text(encoding="utf-8")) - claude = _normalise_whitespace( - (_ROOT / "CLAUDE.md").read_text(encoding="utf-8") - ) - for text, label in ((agents, "AGENTS.md"), (claude, "CLAUDE.md")): + claude = _normalise_whitespace(_claude_rules_text()) + for text, label in ((agents, "AGENTS.md"), (claude, "CLAUDE.md+subagents")): # The anti-pattern bullet that introduces the rule. assert "irrelevant downloads" in text.lower(), ( f"{label} lost the 'irrelevant downloads' anti-pattern" @@ -196,13 +210,13 @@ def test_pruning_irrelevant_downloads_rule_documented(): assert "pdfs/" in text and ".pptx" in text, ( f"{label} lost the concrete pdfs/<key>.pdf + <key>.pptx paths" ) - # The CLAUDE.md canonical reference must also carry the + # The Claude-side canonical reference must also carry the # "Pruning irrelevant downloads" sub-heading + the keep-xlsx note. assert "Pruning irrelevant downloads" in claude, ( - "CLAUDE.md lost the 'Pruning irrelevant downloads' sub-heading" + "CLAUDE.md+subagents lost the 'Pruning irrelevant downloads' sub-heading" ) assert "honest record" in claude, ( - "CLAUDE.md lost the 'keep the aggregate xlsx/bib' rationale" + "CLAUDE.md+subagents lost the 'keep the aggregate xlsx/bib' rationale" ) @@ -215,10 +229,8 @@ def test_url_doi_verification_rule_documented(): see the rule and the concrete audit snippet so the lesson sticks. """ agents = _normalise_whitespace(_AGENTS_MD.read_text(encoding="utf-8")) - claude = _normalise_whitespace( - (_ROOT / "CLAUDE.md").read_text(encoding="utf-8") - ) - for text, label in ((agents, "AGENTS.md"), (claude, "CLAUDE.md")): + claude = _normalise_whitespace(_claude_rules_text()) + for text, label in ((agents, "AGENTS.md"), (claude, "CLAUDE.md+subagents")): # Rule heading assert "URL / DOI verification" in text, ( f"{label} lost the URL/DOI verification rule heading" diff --git a/tests/test_cli.py b/tests/test_cli.py index 02bbf49..64787e1 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -43,7 +43,7 @@ async def _fake_success(collection, out_dir): @pytest.fixture() def patched_pipeline(monkeypatch, sample_papers): - async def fake_run_search(query: Query) -> PaperCollection: + async def fake_run_search(query: Query, **_kwargs) -> PaperCollection: return PaperCollection(query=query, papers=tuple(sample_papers)) async def fake_shutdown() -> None: @@ -92,7 +92,7 @@ def test_cli_rejects_unknown_export(tmp_path, patched_pipeline): def test_cli_no_results_returns_one(tmp_path, monkeypatch): - async def empty_pipeline(query: Query) -> PaperCollection: + async def empty_pipeline(query: Query, **_kwargs) -> PaperCollection: return PaperCollection(query=query, papers=()) async def fake_shutdown() -> None: @@ -221,7 +221,7 @@ def test_cli_source_default_is_multi_source(tmp_path, monkeypatch, sample_papers captured: dict[str, Query] = {} - async def fake_run_search(query: Query) -> PaperCollection: + async def fake_run_search(query: Query, **_kwargs) -> PaperCollection: captured["query"] = query return PaperCollection(query=query, papers=tuple(sample_papers)) @@ -238,11 +238,12 @@ async def fake_shutdown() -> None: assert captured["query"].sources == DEFAULT_SOURCES -def test_cli_top_tier_filter_on_by_default(tmp_path, monkeypatch, sample_papers): - """The CLI must turn on top_tier_only by default; --all-venues disables.""" +def test_cli_top_tier_filter_off_by_default(tmp_path, monkeypatch, sample_papers): + """top_tier_only is OFF by default (broader coverage including IEEE / ACM + workshops); --top-tier-only flips it on.""" captured: dict[str, Query] = {} - async def fake_run_search(query: Query) -> PaperCollection: + async def fake_run_search(query: Query, **_kwargs) -> PaperCollection: captured["query"] = query return PaperCollection(query=query, papers=tuple(sample_papers)) @@ -256,21 +257,21 @@ async def fake_shutdown() -> None: ["--query", "x", "--out", str(tmp_path), "--export", "bib"] ) assert code == 0 - assert captured["query"].top_tier_only is True + assert captured["query"].top_tier_only is False captured.clear() code = cli_module.main( - ["--query", "x", "--all-venues", "--out", str(tmp_path), "--export", "bib"] + ["--query", "x", "--top-tier-only", "--out", str(tmp_path), "--export", "bib"] ) assert code == 0 - assert captured["query"].top_tier_only is False + assert captured["query"].top_tier_only is True def test_cli_default_triggers_pdf_download(tmp_path, monkeypatch, sample_papers): """Default flag set should invoke download_pdfs; --no-pdf disables it.""" calls: list[str] = [] - async def fake_run_search(query: Query) -> PaperCollection: + async def fake_run_search(query: Query, **_kwargs) -> PaperCollection: return PaperCollection(query=query, papers=tuple(sample_papers)) async def fake_shutdown() -> None: @@ -299,7 +300,7 @@ async def fake_download(_collection, _out_dir): def test_cli_no_pdf_flag_skips_download(tmp_path, monkeypatch, sample_papers): calls: list[str] = [] - async def fake_run_search(query: Query) -> PaperCollection: + async def fake_run_search(query: Query, **_kwargs) -> PaperCollection: return PaperCollection(query=query, papers=tuple(sample_papers)) async def fake_shutdown() -> None: @@ -351,7 +352,7 @@ def _build_paper(source_id: str, *, pdf_url: str | None): def _patch_search(monkeypatch, papers): from autopapertoppt.core.models import PaperCollection - async def fake_run_search(query): + async def fake_run_search(query, **_kwargs): return PaperCollection(query=query, papers=tuple(papers)) async def fake_shutdown(): @@ -648,7 +649,7 @@ async def fake_enrich(collection, language=None, model=None): # noqa: ARG001 def _fake_search_with_papers(monkeypatch, sample_papers): - async def fake_run_search(query): + async def fake_run_search(query, **_kwargs): return PaperCollection(query=query, papers=tuple(sample_papers)) async def fake_shutdown(): diff --git a/tests/test_i18n.py b/tests/test_i18n.py index 112b0f4..a420fbd 100644 --- a/tests/test_i18n.py +++ b/tests/test_i18n.py @@ -88,21 +88,21 @@ def test_every_supported_language_has_readme_and_sphinx_tree(): from drifting apart — if a new language is added to one, the test fails loudly until the other is filled in too. - File-name convention: English is the canonical README.md; every other - language uses README.<lang>.md and docs/<lang>/index.rst. The zh-TW - README file uses the historical mixed-case ``zh-TW`` to match the - Languages: navigation links across the repo; everywhere else the - folder/code is lowercase. + File-name convention: English is the canonical ``README.md`` at the + repo root; every other language lives under ``readmes/README.<lang>.md`` + plus its own ``docs/<lang>/index.rst``. The zh-TW README file uses the + historical mixed-case ``zh-TW`` to match the Languages: navigation + links across the repo; everywhere else the folder/code is lowercase. """ readme_overrides = { "en": "README.md", - "zh-tw": "README.zh-TW.md", - "zh-cn": "README.zh-CN.md", + "zh-tw": "readmes/README.zh-TW.md", + "zh-cn": "readmes/README.zh-CN.md", } missing_readme: list[str] = [] missing_sphinx: list[str] = [] for lang in SUPPORTED_LANGUAGES: - readme_name = readme_overrides.get(lang, f"README.{lang}.md") + readme_name = readme_overrides.get(lang, f"readmes/README.{lang}.md") if not (_REPO_ROOT / readme_name).is_file(): missing_readme.append(f"{lang} (expected {readme_name})") sphinx_index = _REPO_ROOT / "docs" / lang / "index.rst" @@ -157,12 +157,12 @@ def test_prune_irrelevant_downloads_rule_in_all_14_languages(): } readme_overrides = { "en": "README.md", - "zh-tw": "README.zh-TW.md", - "zh-cn": "README.zh-CN.md", + "zh-tw": "readmes/README.zh-TW.md", + "zh-cn": "readmes/README.zh-CN.md", } missing: list[str] = [] for lang, marker in markers.items(): - readme_path = _REPO_ROOT / readme_overrides.get(lang, f"README.{lang}.md") + readme_path = _REPO_ROOT / readme_overrides.get(lang, f"readmes/README.{lang}.md") sphinx_path = _REPO_ROOT / "docs" / lang / "index.rst" if not re.search(marker, readme_path.read_text(encoding="utf-8")): missing.append(f"README {lang}: missing prune-irrelevant marker") @@ -210,7 +210,7 @@ def test_zh_tw_files_use_traditional_chinese_vocabulary(): ] zh_tw_paths = [ _REPO_ROOT / "scripts" / "regen_llm_security_batch_zh_tw.py", - _REPO_ROOT / "README.zh-TW.md", + _REPO_ROOT / "readmes" / "README.zh-TW.md", _REPO_ROOT / "docs" / "zh-tw" / "index.rst", ] offenders: list[str] = [] diff --git a/tests/test_mcp_tools.py b/tests/test_mcp_tools.py index c2f9d6a..f488757 100644 --- a/tests/test_mcp_tools.py +++ b/tests/test_mcp_tools.py @@ -36,7 +36,7 @@ def _first_text(result): def test_search_tool(monkeypatch, server, sample_papers): - async def fake_run_search(query): + async def fake_run_search(query, **_kwargs): return PaperCollection(query=query, papers=tuple(sample_papers)) async def fake_shutdown(): @@ -142,12 +142,13 @@ def test_pptx_inspect_and_update_via_mcp(server, sample_papers, tmp_path): def test_list_sources_tool(server, monkeypatch): """list_sources reports every plugin + reflects current env-var state.""" - # Ensure the opt-in plugins are disabled in this test process. + # Clear every gating var so the default-on / opt-in semantics are + # exercised without contamination from the host shell. for var in ( "AUTOPAPERTOPPT_IEEE_API_KEY", - "AUTOPAPERTOPPT_ENABLE_IEEE_SCRAPING", + "AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING", "AUTOPAPERTOPPT_SPRINGER_API_KEY", - "AUTOPAPERTOPPT_ENABLE_SCHOLAR_SCRAPING", + "AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING", ): monkeypatch.delenv(var, raising=False) payload = asyncio.run(_call(server, "list_sources")) @@ -162,11 +163,18 @@ def test_list_sources_tool(server, monkeypatch): # Plugins that need no env var must be enabled. assert names["arxiv"]["enabled"] is True assert names["dblp"]["enabled"] is True - # Plugins gated by env vars must be disabled when those vars are unset. + # IEEE + Scholar are now default-ON (no opt-out env var set). + assert names["ieee"]["enabled"] is True + assert names["ieee"]["opt_out_env_var"] == ["AUTOPAPERTOPPT_DISABLE_IEEE_SCRAPING"] + assert names["scholar"]["enabled"] is True + assert names["scholar"]["opt_out_env_var"] == ["AUTOPAPERTOPPT_DISABLE_SCHOLAR_SCRAPING"] + # Springer still opt-IN — without the API key it is disabled. assert names["springer"]["enabled"] is False - assert names["springer"]["needs_env_var"] == ["AUTOPAPERTOPPT_SPRINGER_API_KEY"] - assert names["scholar"]["enabled"] is False + assert names["springer"]["opt_in_env_var"] == ["AUTOPAPERTOPPT_SPRINGER_API_KEY"] assert "default_sources" in payload + # Default mix now includes scholar (alongside ieee + the others). + assert "scholar" in payload["default_sources"] + assert "ieee" in payload["default_sources"] def test_list_sources_reflects_springer_key(server, monkeypatch): @@ -180,7 +188,7 @@ def test_search_passes_top_tier_and_min_citations(monkeypatch, server, sample_pa """top_tier_only + min_citations flow from the MCP tool into the Query.""" captured = {} - async def fake_run_search(query): + async def fake_run_search(query, **_kwargs): captured["query"] = query return PaperCollection(query=query, papers=tuple(sample_papers)) @@ -210,7 +218,7 @@ def test_search_defaults_to_full_source_mix(monkeypatch, server, sample_papers): captured = {} - async def fake_run_search(query): + async def fake_run_search(query, **_kwargs): captured["query"] = query return PaperCollection(query=query, papers=tuple(sample_papers)) diff --git a/tests/test_oa_resolver.py b/tests/test_oa_resolver.py new file mode 100644 index 0000000..f0c4c82 --- /dev/null +++ b/tests/test_oa_resolver.py @@ -0,0 +1,320 @@ +"""Tests for ``autopapertoppt.core.oa_resolver``.""" + +from __future__ import annotations + +import pytest + +from autopapertoppt.core import oa_resolver +from autopapertoppt.core.models import Paper, PaperCollection, Query + + +def _paper(**overrides) -> Paper: + defaults = { + "source": "openalex", + "source_id": "W123", + "title": "Attention Is All You Need", + "authors": ("Vaswani",), + "year": 2017, + "venue": "NeurIPS", + "abstract": "...", + "url": "https://example.com/abs", + "doi": "10.5555/example", + "arxiv_id": None, + "pdf_url": None, + } + defaults.update(overrides) + return Paper(**defaults) + + +@pytest.fixture(autouse=True) +def _reset_warning_flag(): + """One-shot warning flags reset between tests.""" + oa_resolver._email_warning_emitted = False # noqa: SLF001 + oa_resolver._core_warning_emitted = False # noqa: SLF001 + yield + oa_resolver._email_warning_emitted = False # noqa: SLF001 + oa_resolver._core_warning_emitted = False # noqa: SLF001 + + +def test_arxiv_id_to_pdf_strips_version_suffix(): + assert oa_resolver._arxiv_id_to_pdf("1706.03762") == "https://arxiv.org/pdf/1706.03762.pdf" # noqa: SLF001 + assert oa_resolver._arxiv_id_to_pdf("1706.03762v2") == "https://arxiv.org/pdf/1706.03762.pdf" # noqa: SLF001 + assert oa_resolver._arxiv_id_to_pdf("cs.LG/0001001v1") == "https://arxiv.org/pdf/cs.LG/0001001.pdf" # noqa: SLF001 + assert oa_resolver._arxiv_id_to_pdf("") is None # noqa: SLF001 + assert oa_resolver._arxiv_id_to_pdf(" ") is None # noqa: SLF001 + + +async def test_resolve_uses_arxiv_id_direct_before_unpaywall(monkeypatch): + """If arxiv_id is set, derive the PDF URL directly with no HTTP call.""" + unpaywall_calls: list[str] = [] + + async def fake_unpaywall(doi): + unpaywall_calls.append(doi) + return + + monkeypatch.setattr(oa_resolver, "_query_unpaywall", fake_unpaywall) + + paper = _paper(arxiv_id="1706.03762") + collection = PaperCollection( + query=Query(keywords="x", sources=("openalex",)), + papers=(paper,), + ) + result = await oa_resolver.resolve_oa_pdfs(collection) + assert result.papers[0].pdf_url == "https://arxiv.org/pdf/1706.03762.pdf" + # Unpaywall should NOT have been called — arxiv_id short-circuited. + assert unpaywall_calls == [] + + +async def test_resolve_falls_back_to_s2_when_unpaywall_misses(monkeypatch): + async def fake_unpaywall(_doi): + return None + + async def fake_s2(doi): + assert doi == "10.5555/example" + return "https://semantic-scholar-oa.example/p.pdf" + + monkeypatch.setattr(oa_resolver, "_query_unpaywall", fake_unpaywall) + monkeypatch.setattr(oa_resolver, "_query_semantic_scholar", fake_s2) + + collection = PaperCollection( + query=Query(keywords="x", sources=("openalex",)), + papers=(_paper(),), + ) + result = await oa_resolver.resolve_oa_pdfs(collection) + assert result.papers[0].pdf_url == "https://semantic-scholar-oa.example/p.pdf" + + +async def test_resolve_falls_back_to_core_when_s2_misses(monkeypatch): + async def miss(_doi): + return None + + async def fake_core(doi): + assert doi == "10.5555/example" + return "https://institutional-repo.example/p.pdf" + + monkeypatch.setattr(oa_resolver, "_query_unpaywall", miss) + monkeypatch.setattr(oa_resolver, "_query_semantic_scholar", miss) + monkeypatch.setattr(oa_resolver, "_query_core", fake_core) + + collection = PaperCollection( + query=Query(keywords="x", sources=("openalex",)), + papers=(_paper(),), + ) + result = await oa_resolver.resolve_oa_pdfs(collection) + assert result.papers[0].pdf_url == "https://institutional-repo.example/p.pdf" + + +async def test_core_skipped_silently_when_key_unset(monkeypatch): + monkeypatch.delenv("AUTOPAPERTOPPT_CORE_API_KEY", raising=False) + result = await oa_resolver._query_core("10.x/y") # noqa: SLF001 + assert result is None + assert oa_resolver._core_warning_emitted is True # noqa: SLF001 + + +async def test_s2_cache_skips_repeat_lookups(monkeypatch): + """A DOI looked up once is served from in-process cache the second time.""" + monkeypatch.setattr(oa_resolver, "_S2_CACHE", {"10.x/cached": "https://oa.example/p.pdf"}) + result = await oa_resolver._query_semantic_scholar("10.x/cached") # noqa: SLF001 + assert result == "https://oa.example/p.pdf" + + +async def test_s2_api_key_sent_when_set(monkeypatch): + """When AUTOPAPERTOPPT_S2_API_KEY is set, the resolver attaches x-api-key.""" + monkeypatch.setattr(oa_resolver, "_S2_CACHE", {}) + monkeypatch.setenv("AUTOPAPERTOPPT_S2_API_KEY", "test-key") + + captured = {} + + class FakeResponse: + status_code = 200 + + def json(self): + return {"openAccessPdf": {"url": "https://s2.example/p.pdf"}} + + class FakeClient: + async def get(self, url, params=None, headers=None): + captured["headers"] = headers + return FakeResponse() + + async def fake_get_client(_name): + return FakeClient() + + monkeypatch.setattr(oa_resolver, "get_client", fake_get_client) + + result = await oa_resolver._query_semantic_scholar("10.x/with-key") # noqa: SLF001 + assert result == "https://s2.example/p.pdf" + assert captured["headers"] == {"x-api-key": "test-key"} + + +async def test_resolve_returns_unchanged_when_all_papers_have_pdf_url(): + collection = PaperCollection( + query=Query(keywords="x", sources=("openalex",)), + papers=(_paper(pdf_url="https://example.com/p.pdf"),), + ) + result = await oa_resolver.resolve_oa_pdfs(collection) + assert result.papers[0].pdf_url == "https://example.com/p.pdf" + # Early-exit returns the same instance — no lookups happened. + assert result is collection + + +async def test_resolve_fills_pdf_url_from_unpaywall(monkeypatch): + async def fake_unpaywall(doi: str) -> str | None: + assert doi == "10.5555/example" + return "https://oa-mirror.example/paper.pdf" + + async def fake_arxiv(_paper): + pytest.fail("arXiv fallback should not run when Unpaywall hits") + + monkeypatch.setattr(oa_resolver, "_query_unpaywall", fake_unpaywall) + monkeypatch.setattr(oa_resolver, "_query_arxiv_title", fake_arxiv) + + collection = PaperCollection( + query=Query(keywords="x", sources=("openalex",)), + papers=(_paper(),), + ) + result = await oa_resolver.resolve_oa_pdfs(collection) + assert result.papers[0].pdf_url == "https://oa-mirror.example/paper.pdf" + + +async def test_resolve_falls_back_to_arxiv_when_unpaywall_misses(monkeypatch): + async def fake_unpaywall(_doi: str) -> str | None: + return None + + async def fake_arxiv(_paper): + return "https://arxiv.org/pdf/1706.03762" + + monkeypatch.setattr(oa_resolver, "_query_unpaywall", fake_unpaywall) + monkeypatch.setattr(oa_resolver, "_query_arxiv_title", fake_arxiv) + + collection = PaperCollection( + query=Query(keywords="x", sources=("openalex",)), + papers=(_paper(),), + ) + result = await oa_resolver.resolve_oa_pdfs(collection) + assert result.papers[0].pdf_url == "https://arxiv.org/pdf/1706.03762" + + +async def test_resolve_passes_through_when_no_doi_and_no_arxiv_hit(monkeypatch): + async def fake_arxiv(_paper): + return None + + monkeypatch.setattr(oa_resolver, "_query_arxiv_title", fake_arxiv) + + collection = PaperCollection( + query=Query(keywords="x", sources=("openalex",)), + papers=(_paper(doi=None),), + ) + result = await oa_resolver.resolve_oa_pdfs(collection) + assert result.papers[0].pdf_url is None + + +async def test_unpaywall_skipped_silently_when_email_unset(monkeypatch): + monkeypatch.delenv("AUTOPAPERTOPPT_CONTACT_EMAIL", raising=False) + result = await oa_resolver._query_unpaywall("10.x/y") # noqa: SLF001 + assert result is None + # And the one-shot warning got emitted. + assert oa_resolver._email_warning_emitted is True # noqa: SLF001 + + +async def test_unpaywall_skip_does_not_double_warn(monkeypatch): + """The one-shot flag prevents repeat warnings within a single process.""" + monkeypatch.delenv("AUTOPAPERTOPPT_CONTACT_EMAIL", raising=False) + call_count = 0 + + def counting_warning(*_args, **_kwargs): + nonlocal call_count + call_count += 1 + + monkeypatch.setattr(oa_resolver._LOG, "warning", counting_warning) # noqa: SLF001 + await oa_resolver._query_unpaywall("10.x/y") # noqa: SLF001 + await oa_resolver._query_unpaywall("10.x/z") # noqa: SLF001 + await oa_resolver._query_unpaywall("10.x/w") # noqa: SLF001 + assert call_count == 1 + + +async def test_normalise_title_strips_punctuation_and_case(): + norm = oa_resolver._normalise_title # noqa: SLF001 + assert norm("Attention Is All You Need") == "attentionisallyouneed" + assert norm("Attention: Is All You Need!") == "attentionisallyouneed" + assert norm("Attention is all you need.") == "attentionisallyouneed" + + +async def test_arxiv_fallback_skips_arxiv_sourced_papers(): + paper = _paper(source="arxiv", pdf_url=None) + result = await oa_resolver._query_arxiv_title(paper) # noqa: SLF001 + assert result is None + + +async def test_arxiv_fallback_matches_only_exact_normalised_title(monkeypatch): + """Title search should NOT accept loosely-similar titles.""" + from unittest.mock import MagicMock + + async def fake_search(query): + # Return a paper whose title overlaps but isn't a real match. + return [ + _paper( + title="Attention is all you need for a totally different topic", + pdf_url="https://arxiv.org/pdf/9999.99999", + ) + ] + + fake_fetcher = MagicMock() + fake_fetcher.search = fake_search + + def fake_load(_name): + return fake_fetcher + + monkeypatch.setattr( + "autopapertoppt.fetchers.base.load_fetcher", fake_load + ) + result = await oa_resolver._query_arxiv_title(_paper()) # noqa: SLF001 + assert result is None + + +async def test_arxiv_fallback_accepts_exact_match(monkeypatch): + from unittest.mock import MagicMock + + async def fake_search(query): + return [ + _paper( + title="Attention Is All You Need", + pdf_url="https://arxiv.org/pdf/1706.03762", + ) + ] + + fake_fetcher = MagicMock() + fake_fetcher.search = fake_search + + def fake_load(_name): + return fake_fetcher + + monkeypatch.setattr( + "autopapertoppt.fetchers.base.load_fetcher", fake_load + ) + result = await oa_resolver._query_arxiv_title(_paper()) # noqa: SLF001 + assert result == "https://arxiv.org/pdf/1706.03762" + + +async def test_arxiv_fallback_rejects_non_https(): + """Defence in depth: even if arxiv returned an http:// URL we don't keep it.""" + # The HTTPS-only transport would reject the download later anyway, + # but the resolver should not stash a known-bad URL into the Paper. + from unittest.mock import MagicMock + + async def fake_search(query): + return [ + _paper( + title="Attention Is All You Need", + pdf_url="http://arxiv.org/pdf/1706.03762", + ) + ] + + fake_fetcher = MagicMock() + fake_fetcher.search = fake_search + + import pytest as _pytest # local import — monkeypatch is fn-level + + with _pytest.MonkeyPatch().context() as mp: + mp.setattr("autopapertoppt.fetchers.base.load_fetcher", lambda _: fake_fetcher) + result = await oa_resolver._query_arxiv_title(_paper()) # noqa: SLF001 + assert result is None diff --git a/tests/test_webrunner_pdf.py b/tests/test_webrunner_pdf.py new file mode 100644 index 0000000..0f237f2 --- /dev/null +++ b/tests/test_webrunner_pdf.py @@ -0,0 +1,129 @@ +"""Tests for ``autopapertoppt.fetchers.webrunner_pdf``.""" + +from __future__ import annotations + +import pytest + +from autopapertoppt.fetchers import webrunner_pdf + + +@pytest.mark.parametrize( + "url", + [ + "https://dl.acm.org/doi/pdf/10.1145/3411764.3445005", + "https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10965643", + "https://link.springer.com/content/pdf/10.1007/s11042.pdf", + "https://www.sciencedirect.com/science/article/pii/X.pdf", + "https://onlinelibrary.wiley.com/doi/pdfdirect/10.1002/x.pdf", + "https://academic.oup.com/journal/article/X/Y/pdf", + "https://www.nature.com/articles/x.pdf", + "https://www.science.org/doi/pdf/10.1126/x", + ], +) +def test_should_use_webrunner_matches_paywalled_publishers(url): + assert webrunner_pdf.should_use_webrunner(url) is True + + +@pytest.mark.parametrize( + "url", + [ + "https://arxiv.org/pdf/1706.03762.pdf", + "https://export.arxiv.org/pdf/2401.08741.pdf", + "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1234/pdf/x.pdf", + "https://www.example.com/paper.pdf", + "https://oa-repo.university.edu/files/x.pdf", + ], +) +def test_should_use_webrunner_skips_open_access_hosts(url): + assert webrunner_pdf.should_use_webrunner(url) is False + + +def test_should_use_webrunner_handles_garbage_url(): + assert webrunner_pdf.should_use_webrunner("") is False + assert webrunner_pdf.should_use_webrunner("not-a-url") is False + + +def test_is_available_skipped_when_disable_env_set(monkeypatch): + monkeypatch.setenv("AUTOPAPERTOPPT_DISABLE_WEBRUNNER", "1") + assert webrunner_pdf.is_available() is False + + +def test_is_available_returns_true_when_selenium_present(monkeypatch): + monkeypatch.delenv("AUTOPAPERTOPPT_DISABLE_WEBRUNNER", raising=False) + # selenium is in [dev] extras so it's importable in the test venv. + assert webrunner_pdf.is_available() is True + + +async def test_pdf_download_routes_paywalled_through_webrunner(monkeypatch, tmp_path): + """The PDF downloader pipeline routes paywalled URLs via WebRunner.""" + from autopapertoppt.core import pdf_download + from autopapertoppt.core.models import Paper, PaperCollection, Query + + captured: dict[str, object] = {} + + async def fake_browser_download(url, target): + captured["url"] = url + captured["target"] = target + # Write a fake PDF so the persistence check passes. + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(b"%PDF-1.4\n...fake body...\n%%EOF") + return True + + monkeypatch.setattr(webrunner_pdf, "is_available", lambda: True) + monkeypatch.setattr(webrunner_pdf, "download_via_browser", fake_browser_download) + + paper = Paper( + source="acm", source_id="X", + title="ACM paper", authors=("A",), year=2025, + venue="ACM CCS", abstract="...", + url="https://dl.acm.org/doi/10.1145/X", + doi="10.1145/X", arxiv_id=None, + pdf_url="https://dl.acm.org/doi/pdf/10.1145/X", + ) + collection = PaperCollection( + query=Query(keywords="x", sources=("acm",)), + papers=(paper,), + ) + results = await pdf_download.download_pdfs(collection, tmp_path) + assert len(results) == 1 + assert results[0].skipped_reason is None + assert results[0].path is not None + assert results[0].path.exists() + assert captured["url"] == "https://dl.acm.org/doi/pdf/10.1145/X" + + +async def test_pdf_download_skips_webrunner_for_arxiv(monkeypatch, tmp_path): + """arXiv PDFs bypass WebRunner — httpx works fine on arXiv.""" + from autopapertoppt.core import pdf_download + from autopapertoppt.core.models import Paper, PaperCollection, Query + + async def fail_browser(_url, _target): + pytest.fail("WebRunner should not be called for arXiv URLs") + + monkeypatch.setattr(webrunner_pdf, "is_available", lambda: True) + monkeypatch.setattr(webrunner_pdf, "download_via_browser", fail_browser) + + # Stub the httpx fetch so we don't hit the real network. + async def fake_fetch_and_validate(_paper, target, _key): + target.write_bytes(b"%PDF-1.4\nfrom arxiv\n%%EOF") + return pdf_download.PdfDownloadResult( + paper_key=_key, path=target, skipped_reason=None, + ) + + monkeypatch.setattr(pdf_download, "_fetch_and_validate", fake_fetch_and_validate) + + paper = Paper( + source="arxiv", source_id="1706.03762", + title="Attention", authors=("V",), year=2017, + venue=None, abstract="...", + url="https://arxiv.org/abs/1706.03762", + doi=None, arxiv_id="1706.03762", + pdf_url="https://arxiv.org/pdf/1706.03762.pdf", + ) + collection = PaperCollection( + query=Query(keywords="x", sources=("arxiv",)), + papers=(paper,), + ) + results = await pdf_download.download_pdfs(collection, tmp_path) + assert results[0].skipped_reason is None + assert results[0].path is not None