Skip to content

Validate LLM and judge model can be accessed publicly / using token provided #211

@linear

Description

@linear

When creating an LLM or an LLM behavior eval, ensure that the LLM and / or judge model can be accessed either publicly (if no respective token is provided) or using the provided token. If metadata can be provided in the error to the user such as "gated-only model, please provide a HuggingFace token" or "private model, please provide a HuggingFace model" to make it clear why a token must be provided, then please do so and if not, please provide a generic message which suggests either potential option or that the model ID was incorrect.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions