Skip to content

fix: use runtime factory for eval entrypoint discovery and show usage help#1398

Open
cristipufu wants to merge 1 commit intomainfrom
fix/eval-discovery-usage-help
Open

fix: use runtime factory for eval entrypoint discovery and show usage help#1398
cristipufu wants to merge 1 commit intomainfrom
fix/eval-discovery-usage-help

Conversation

@cristipufu
Copy link
Member

@cristipufu cristipufu commented Mar 2, 2026

Summary

  • Replace hardcoded auto_discover_entrypoint() (uipath.json-only) with runtime_factory.discover_entrypoints() which supports all runtimes (langgraph, llama_index, etc.)
  • When auto-discovery fails (multiple entrypoints or eval sets), show a clean usage message listing available options with example commands instead of a ValueError traceback
  • Remove unused auto_discover_entrypoint and auto_discover_eval_set functions

Test plan

  • Added 7 new tests in tests/cli/eval/test_eval_discovery.py
  • Multiple entrypoints shows usage help with all available entrypoints and eval sets
  • Multiple eval sets shows usage help with full relative paths
  • Single entrypoint + eval set auto-discovers without showing help
  • No entrypoints shows helpful guidance message
  • Explicit entrypoint still triggers eval set discovery
  • All 1514 existing tests pass

🤖 Generated with Claude Code

Development Package

  • Use uipath pack --nolock to get the latest dev build from this PR (requires version range).
  • Add this package as a dependency in your pyproject.toml:
[project]
dependencies = [
  # Exact version:
  "uipath==2.10.4.dev1013985129",

  # Any version from PR
  "uipath>=2.10.4.dev1013980000,<2.10.4.dev1013990000"
]

[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

[tool.uv.sources]
uipath = { index = "testpypi" }

[tool.uv]
override-dependencies = [
    "uipath>=2.10.4.dev1013980000,<2.10.4.dev1013990000",
]

@github-actions github-actions bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository labels Mar 2, 2026
@cristipufu cristipufu requested a review from Copilot March 2, 2026 16:04
@cristipufu cristipufu self-assigned this Mar 2, 2026
@cristipufu cristipufu added the build:dev Create a dev build from the pr label Mar 2, 2026
@cristipufu cristipufu requested a review from radu-mocanu March 2, 2026 16:05
@cristipufu cristipufu force-pushed the fix/eval-discovery-usage-help branch from e9916b3 to cb8a5b8 Compare March 2, 2026 16:06

This comment was marked as outdated.

@cristipufu cristipufu force-pushed the fix/eval-discovery-usage-help branch from cb8a5b8 to e9e47ca Compare March 2, 2026 16:57
… help

Replace hardcoded uipath.json-only entrypoint discovery with
runtime_factory.discover_entrypoints() which supports all runtimes
(langgraph, llama_index, etc). When auto-discovery fails (multiple
entrypoints or eval sets), show a clean usage message listing available
options instead of a ValueError traceback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@cristipufu cristipufu force-pushed the fix/eval-discovery-usage-help branch from e9e47ca to 24b1ac5 Compare March 3, 2026 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build:dev Create a dev build from the pr test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants