openai-markdown-docs/api-reference/evals/create.md at master · razmser/openai-markdown-docs

Create eval

post /evals

Create the structure of an evaluation that can be used to test a model's performance. An evaluation is a set of testing criteria and the config for a data source, which dictates the schema of the data used in the evaluation. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources. For more information, see the Evals guide.

Body Parameters

data_source_config: object { item_schema, type, include_sample_schema } or object { type, metadata } or object { type, metadata }

The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.
- CustomDataSourceConfig object { item_schema, type, include_sample_schema }
  
  A CustomDataSourceConfig object that defines the schema for the data source used for the evaluation runs. This schema is used to define the shape of the data that will be:
  - Used to define your testing criteria and
  - What data is required when creating a run
  - item_schema: map[unknown]
    
    The json schema for each row in the data source.
  - type: "custom"
    
    The type of data source. Always custom.
    - "custom"
  - include_sample_schema: optional boolean
    
    Whether the eval should expect you to populate the sample namespace (ie, by generating responses off of your data source)
- LogsDataSourceConfig object { type, metadata }
  
  A data source config which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc.
  - type: "logs"
    
    The type of data source. Always logs.
    - "logs"
  - metadata: optional map[unknown]
    
    Metadata filters for the logs data source.
- StoredCompletionsDataSourceConfig object { type, metadata }
  
  Deprecated in favor of LogsDataSourceConfig.
  - type: "stored_completions"
    
    The type of data source. Always stored_completions.
    - "stored_completions"
  - metadata: optional map[unknown]
    
    Metadata filters for the stored completions data source.
testing_criteria: array of object { input, labels, model, 3 more } or StringCheckGrader or TextSimilarityGrader or 2 more

A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like {{item.variable_name}}. To reference the model's output, use the sample namespace (ie, {{sample.output_text}}).
- LabelModelGrader object { input, labels, model, 3 more }
  
  A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
  - input: array of object { content, role } or object { content, role, type }
    
    A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.
    - SimpleInputMessage object { content, role }
      - content: string
        
        The content of the message.
      - role: string
        
        The role of the message (e.g. "system", "assistant", "user").
    - EvalMessageObject object { content, role, type }
      
      A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.
      - content: string or ResponseInputText or object { text, type } or 3 more
        
        Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
        
        TextInput = string
        
        A text input to the model.
        
        ResponseInputText object { text, type }
        
        A text input to the model.
        
        text: string
        
        The text input to the model.
        
        type: "input_text"
        
        The type of the input item. Always input_text.
        
        "input_text"
        
        OutputText object { text, type }
        
        A text output from the model.
        
        text: string
        
        The text output from the model.
        
        type: "output_text"
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        InputImage object { image_url, type, detail }
        
        An image input block used within EvalItem content arrays.
        
        image_url: string
        
        The URL of the image input.
        
        type: "input_image"
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: optional string
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        ResponseInputAudio object { input_audio, type }
        
        An audio input to the model.
        
        input_audio: object { data, format }
        
        data: string
        
        Base64-encoded audio data.
        
        format: "mp3" or "wav"
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        "mp3"
        
        "wav"
        
        type: "input_audio"
        
        The type of the input item. Always input_audio.
        
        "input_audio"
        
        GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        TextInput = string
        
        A text input to the model.
        
        ResponseInputText object { text, type }
        
        A text input to the model.
        
        OutputText object { text, type }
        
        A text output from the model.
        
        text: string
        
        The text output from the model.
        
        type: "output_text"
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        InputImage object { image_url, type, detail }
        
        An image input block used within EvalItem content arrays.
        
        image_url: string
        
        The URL of the image input.
        
        type: "input_image"
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: optional string
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        ResponseInputAudio object { input_audio, type }
        
        An audio input to the model.
      - role: "user" or "assistant" or "system" or "developer"
        
        The role of the message input. One of user, assistant, system, or developer.
        
        "user"
        
        "assistant"
        
        "system"
        
        "developer"
      - type: optional "message"
        
        The type of the message input. Always message.
        
        "message"
  - labels: array of string
    
    The labels to classify to each item in the evaluation.
  - model: string
    
    The model to use for the evaluation. Must support structured outputs.
  - name: string
    
    The name of the grader.
  - passing_labels: array of string
    
    The labels that indicate a passing result. Must be a subset of labels.
  - type: "label_model"
    
    The object type, which is always label_model.
    - "label_model"
- StringCheckGrader object { input, name, operation, 2 more }
  
  A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
  - input: string
    
    The input text. This may include template strings.
  - name: string
    
    The name of the grader.
  - operation: "eq" or "ne" or "like" or "ilike"
    
    The string check operation to perform. One of eq, ne, like, or ilike.
    - "eq"
    - "ne"
    - "like"
    - "ilike"
  - reference: string
    
    The reference text. This may include template strings.
  - type: "string_check"
    
    The object type, which is always string_check.
    - "string_check"
- TextSimilarityGrader = TextSimilarityGrader
  
  A TextSimilarityGrader object which grades text based on similarity metrics.
  - pass_threshold: number
    
    The threshold for the score.
- PythonGrader = PythonGrader
  
  A PythonGrader object that runs a python script on the input.
  - pass_threshold: optional number
    
    The threshold for the score.
- ScoreModelGrader = ScoreModelGrader
  
  A ScoreModelGrader object that uses a model to assign a score to the input.
  - pass_threshold: optional number
    
    The threshold for the score.
metadata: optional Metadata

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
name: optional string

The name of the evaluation.

Returns

id: string

Unique identifier for the evaluation.
created_at: number

The Unix timestamp (in seconds) for when the eval was created.
data_source_config: EvalCustomDataSourceConfig or object { schema, type, metadata } or EvalStoredCompletionsDataSourceConfig

Configuration of data sources used in runs of the evaluation.
- EvalCustomDataSourceConfig object { schema, type }
  
  A CustomDataSourceConfig which specifies the schema of your item and optionally sample namespaces. The response schema defines the shape of the data that will be:
  - Used to define your testing criteria and
  - What data is required when creating a run
  - schema: map[unknown]
    
    The json schema for the run data source items. Learn how to build JSON schemas here.
  - type: "custom"
    
    The type of data source. Always custom.
    - "custom"
- LogsDataSourceConfig object { schema, type, metadata }
  
  A LogsDataSourceConfig which specifies the metadata property of your logs query. This is usually metadata like usecase=chatbot or prompt-version=v2, etc. The schema returned by this data source config is used to defined what variables are available in your evals. item and sample are both defined when using this data source config.
  - schema: map[unknown]
    
    The json schema for the run data source items. Learn how to build JSON schemas here.
  - type: "logs"
    
    The type of data source. Always logs.
    - "logs"
  - metadata: optional Metadata
    
    Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
    
    Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
- EvalStoredCompletionsDataSourceConfig object { schema, type, metadata }
  
  Deprecated in favor of LogsDataSourceConfig.
  - schema: map[unknown]
    
    The json schema for the run data source items. Learn how to build JSON schemas here.
  - type: "stored_completions"
    
    The type of data source. Always stored_completions.
    - "stored_completions"
  - metadata: optional Metadata
    
    Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.
    
    Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
metadata: Metadata

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.
name: string

The name of the evaluation.
object: "eval"

The object type.
- "eval"
testing_criteria: array of LabelModelGrader or StringCheckGrader or TextSimilarityGrader or 2 more

A list of testing criteria.
- LabelModelGrader object { input, labels, model, 3 more }
  
  A LabelModelGrader object which uses a model to assign labels to each item in the evaluation.
  - input: array of object { content, role, type }
    - content: string or ResponseInputText or object { text, type } or 3 more
      
      Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.
      - TextInput = string
        
        A text input to the model.
      - ResponseInputText object { text, type }
        
        A text input to the model.
        
        text: string
        
        The text input to the model.
        
        type: "input_text"
        
        The type of the input item. Always input_text.
        
        "input_text"
      - OutputText object { text, type }
        
        A text output from the model.
        
        text: string
        
        The text output from the model.
        
        type: "output_text"
        
        The type of the output text. Always output_text.
        
        "output_text"
      - InputImage object { image_url, type, detail }
        
        An image input block used within EvalItem content arrays.
        
        image_url: string
        
        The URL of the image input.
        
        type: "input_image"
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: optional string
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
      - ResponseInputAudio object { input_audio, type }
        
        An audio input to the model.
        
        input_audio: object { data, format }
        
        data: string
        
        Base64-encoded audio data.
        
        format: "mp3" or "wav"
        
        The format of the audio data. Currently supported formats are mp3 and wav.
        
        "mp3"
        
        "wav"
        
        type: "input_audio"
        
        The type of the input item. Always input_audio.
        
        "input_audio"
      - GraderInputs = array of string or ResponseInputText or object { text, type } or 2 more
        
        A list of inputs, each of which may be either an input text, output text, input image, or input audio object.
        
        TextInput = string
        
        A text input to the model.
        
        ResponseInputText object { text, type }
        
        A text input to the model.
        
        OutputText object { text, type }
        
        A text output from the model.
        
        text: string
        
        The text output from the model.
        
        type: "output_text"
        
        The type of the output text. Always output_text.
        
        "output_text"
        
        InputImage object { image_url, type, detail }
        
        An image input block used within EvalItem content arrays.
        
        image_url: string
        
        The URL of the image input.
        
        type: "input_image"
        
        The type of the image input. Always input_image.
        
        "input_image"
        
        detail: optional string
        
        The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.
        
        ResponseInputAudio object { input_audio, type }
        
        An audio input to the model.
    - role: "user" or "assistant" or "system" or "developer"
      
      The role of the message input. One of user, assistant, system, or developer.
      - "user"
      - "assistant"
      - "system"
      - "developer"
    - type: optional "message"
      
      The type of the message input. Always message.
      - "message"
  - labels: array of string
    
    The labels to assign to each item in the evaluation.
  - model: string
    
    The model to use for the evaluation. Must support structured outputs.
  - name: string
    
    The name of the grader.
  - passing_labels: array of string
    
    The labels that indicate a passing result. Must be a subset of labels.
  - type: "label_model"
    
    The object type, which is always label_model.
    - "label_model"
- StringCheckGrader object { input, name, operation, 2 more }
  
  A StringCheckGrader object that performs a string comparison between input and reference using a specified operation.
  - input: string
    
    The input text. This may include template strings.
  - name: string
    
    The name of the grader.
  - operation: "eq" or "ne" or "like" or "ilike"
    
    The string check operation to perform. One of eq, ne, like, or ilike.
    - "eq"
    - "ne"
    - "like"
    - "ilike"
  - reference: string
    
    The reference text. This may include template strings.
  - type: "string_check"
    
    The object type, which is always string_check.
    - "string_check"
- TextSimilarityGrader = TextSimilarityGrader
  
  A TextSimilarityGrader object which grades text based on similarity metrics.
  - pass_threshold: number
    
    The threshold for the score.
- PythonGrader = PythonGrader
  
  A PythonGrader object that runs a python script on the input.
  - pass_threshold: optional number
    
    The threshold for the score.
- ScoreModelGrader = ScoreModelGrader
  
  A ScoreModelGrader object that uses a model to assign a score to the input.
  - pass_threshold: optional number
    
    The threshold for the score.

Example

curl https://api.openai.com/v1/evals \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -d '{
          "data_source_config": {
            "item_schema": {
              "foo": "bar"
            },
            "type": "custom"
          },
          "testing_criteria": [
            {
              "input": [
                {
                  "content": "content",
                  "role": "role"
                }
              ],
              "labels": [
                "string"
              ],
              "model": "model",
              "name": "name",
              "passing_labels": [
                "string"
              ],
              "type": "label_model"
            }
          ]
        }'

Response

{
  "id": "id",
  "created_at": 0,
  "data_source_config": {
    "schema": {
      "foo": "bar"
    },
    "type": "custom"
  },
  "metadata": {
    "foo": "string"
  },
  "name": "Chatbot effectiveness Evaluation",
  "object": "eval",
  "testing_criteria": [
    {
      "input": [
        {
          "content": "string",
          "role": "user",
          "type": "message"
        }
      ],
      "labels": [
        "string"
      ],
      "model": "model",
      "name": "name",
      "passing_labels": [
        "string"
      ],
      "type": "label_model"
    }
  ]
}

Example

curl https://api.openai.com/v1/evals \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "Sentiment",
        "data_source_config": {
          "type": "stored_completions",
          "metadata": {
              "usecase": "chatbot"
          }
        },
        "testing_criteria": [
          {
            "type": "label_model",
            "model": "o3-mini",
            "input": [
              {
                "role": "developer",
                "content": "Classify the sentiment of the following statement as one of 'positive', 'neutral', or 'negative'"
              },
              {
                "role": "user",
                "content": "Statement: {{item.input}}"
              }
            ],
            "passing_labels": [
              "positive"
            ],
            "labels": [
              "positive",
              "neutral",
              "negative"
            ],
            "name": "Example label grader"
          }
        ]
      }'

Response

{
  "object": "eval",
  "id": "eval_67b7fa9a81a88190ab4aa417e397ea21",
  "data_source_config": {
    "type": "stored_completions",
    "metadata": {
      "usecase": "chatbot"
    },
    "schema": {
      "type": "object",
      "properties": {
        "item": {
          "type": "object"
        },
        "sample": {
          "type": "object"
        }
      },
      "required": [
        "item",
        "sample"
      ]
  },
  "testing_criteria": [
    {
      "name": "Example label grader",
      "type": "label_model",
      "model": "o3-mini",
      "input": [
        {
          "type": "message",
          "role": "developer",
          "content": {
            "type": "input_text",
            "text": "Classify the sentiment of the following statement as one of positive, neutral, or negative"
          }
        },
        {
          "type": "message",
          "role": "user",
          "content": {
            "type": "input_text",
            "text": "Statement: {{item.input}}"
          }
        }
      ],
      "passing_labels": [
        "positive"
      ],
      "labels": [
        "positive",
        "neutral",
        "negative"
      ]
    }
  ],
  "name": "Sentiment",
  "created_at": 1740110490,
  "metadata": {
    "description": "An eval for sentiment analysis"
  }
}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create eval

Body Parameters

Returns

Example

Response

Example

Response

FilesExpand file tree

create.md

Latest commit

History

create.md

File metadata and controls

Create eval

Body Parameters

Returns

Example

Response

Example

Response