No public methods to get tool steps as they come in Chat.__call__

[nev-ball]

When using tools with a Chat or AsyncChat class, it would be useful to have a publicly documented way of getting the LLM and tool responses at each step, rather than just when the whole call has finished. This would allow e.g. providing interim feedback to a user for longer running/multi-step calls.

This can currently be done by explicitly using the undocumented Chat._call method - so all that is needed is a public method around that and documentation.

[jph00]

Could you show us a code snippet example of a use case you'd like to support?

[nev-ball]

The use case would be to provide updates to a user during long running queries. E.g. An LLM is given a tool for querying a large db, and then asked questions about that data.

Currently we can run such a query using (Async)Chat.__call__, but will receive nothing back until the LLM has completely finished all tool calls and formulated a response (which could take e.g. 30s to a couple minutes).

The proposal would be to expose a method (called e.g. "tool_loop") that is just a wrapper around (Async)Chat._call that would output responses as they came back to give a user a regular sense of progress and confidence that their query is actually still running.

Example Code snippet:

from typing import Iterable

def db_query(query_term: str, date: Optional[str]="", ...):
    """Query to large db"""
    ...

def run_query(qry: str <Question relating to data in the db e.g. "What product sold the most last Saturday?">) -> Iterable[str]:
    """Inputs a query to an LLM with access to the db. Caller passes interim response back to the user as they come in"""
    chat = Chat(
        model=...,
        sp="Whenever using a tool, please output a message explaining what you are doing",
        tools=[db_query]
    )

    for resp in chat.tool_loop(msg=[qry], max_steps=5):
        if isinstance(resp, dict): # this is what currently comes back when resp is a tool response
            yield "DB query complete, processing data"

        elif resp.choices[0].finish_reason == "tool_calls":
            yield f"Please wait. Querying DB: {resp.choices[0].message.content}"

        elif resp.choices[0].finish_reason == "stop":
           yield resp.choices[0].message.content

        else:
            ... # handle error cases etc.