Support **kwargs

[evanunderscore]

While I can't see a good way of passing arbitrary keyword arguments, users should absolutely be able to pass documented keyword arguments. defopt.run is itself an example of a function that accepts **kwargs for Python 2 compatibility but expects at most argv to be specified.

[evanunderscore]

Perhaps this should only be supported if the function signature is overridden in the docstring as per this example. As a side note, I should update defopt.run to do this.

[evanunderscore]

Everything currently maps to the function signature so it's a bit of work to start adding extra parameters. Can look into this again if someone has a use case for it.

[AbeDillon]

I just left huge comment here, but here is the part that may be relevant to this enhancement:

filter_kwargs is also pretty simple, but it relies on some of the internal workings of defopt, so it makes a better candidate for an enhancement to defopt rather than a stand-alone tool:

import defopt

def filter_kwargs(argv, func, short=None):
    """
    pre-filter the given argument vector such that only valid arguments remain
    return the invalid arguments as a dictionary of pairs.

    Args:
        argv (List[str]):
        func (callable):
        short (Dict[str, str]):

    Returns (Dict[str, str]):
    """
    short = short or {}
    parser = defopt.ArgumentParser(formatter_class=defopt._Formatter)
    parser.set_defaults()
    defopt._populate_parser(func, parser, None, short)
    _, unkonwns = parser.parse_known_args(argv)
    if len(unkonwns) % 2:
        raise ValueError(
            "odd number of remaining args preventing proper key-value pairing: %s", (unknowns,))
    kwargs = {k.lstrip("-").replace("-", "_"): v
              for k, v in zip(unkonwns[::2], unkonwns[1::2])}
    for arg in unkonwns:
        argv.remove(arg)
    return kwargs

In order to attempt casting kwargs to reasonable types, I also use a helper function: yamlize. It converts the Dict[str, str] to a yaml-loadable representation so that the yaml parser can guess the types:

import yaml

def yamlize(d):
    """
    Treat a dict mapping strings to strings as though it was defined in a yaml file so that the
    values can be converted to objects according to the yaml parser.

    Args:
        d (Dict[str, str]): a dict of mapping keys to serialized values

    Returns (Dic[str, Any]): the yaml-interpreted version of d

    Example:
        >>> d = {"a": "123", "b": "hello"}
        >>> yd = yamlize(d)
        >>> assert isinstance(yd['a'], int)
        >>> assert isinstance(yd['b'], str)

    """
    s = "\n".join("%s: %s" % i for i in d.items())
    return yaml.load(s)

[evanunderscore]

yamlize makes me feel a little uneasy, since it puts the responsibility on the user to enter arguments in a particular way in order to get appropriate types, rather than on the developer to specify them in advance.

filter_kwargs looks simple enough, but I'll have to read through #7 a few more times to fully understand your use case. I originally wasn't intending on allowing unrestricted keyword arguments.

[AbeDillon]

I think I agree with you on yamlize. It provides simplicity at the expense of explicitness. It kind-of works in my use-case where any **kwargs are assumed to override fields in a YAML config file, but that's a special enough case that it doesn't need to go into defopt.

The simplest solution might be to pack up the kwargs into a Dict[str, str] like filter_kwargs does and allow the developer to specify a function for converting that to a Dict[str, object]:

def main(**kwargs):
    ...

if __name__ == "__main__":
    defopt.run(main, kwarg_parser=yamlize)

[evanunderscore]

I've been thinking about filter_kwargs, and I think the thing that worries me about it is it starts to stray beyond the capabilities of argparse, which I'd rather avoid if I can. Perhaps a decent compromise would be to make some equivalent of _populate_parser public to allow you to do this in your own code safely?

[anntzer]

I think this is a reasonable feature request, essentially requiring the use of parse_known_args and then "manually" parsing the remainder args into --flag value (or --flag value1 value2 for pairs -- variadic cases would probably be a pain because it becomes impossible to know whether --foo a --bar is {"foo": ["a", "--bar"]} or {"foo": ["a"], "bar": []}). (If there are some subcommands using **kwargs and others not, we could manually check whether there are unknown kwargs passed to a kwargsless subcommand and manually error out in that case.)

The yamlization part seems independent and could be done by annotating **kwargs (or any other arg, in fact) with a pseudo-type yaml_load_string(s: str) -> ... that returns the yaml-loaded value.

I'll leave the implementation to anyone who has a real use for this, though.

[lggruspe]

How about something like this? No filtering needed.

 from argparse import ArgumentParser
                                                                   
 def func(**kwargs: float):                                        
     return kwargs
                                                                   
 parser = ArgumentParser()
 parser.add_argument("--kwargs", nargs="*", default=[])            
 args = parser.parse_args()                                        
 
 def convert_dict(m: list[str]) -> dict[str, float]:
     assert len(m) % 2 == 0
     return dict(zip(m[::2], map(float, m[1::2])))
     
 kwargs = convert_dict(args.kwargs)
 print(kwargs, func(**kwargs))                                

The command would be entered as --kwargs foo 1 bar 2. It still wouldn't work for types like dict[str, list[int]], but it should be possible to make it work for kwargs types with "fixed length."
Ex:

# ex: --kwargs foo 1 2 bar 3 4
def func(**kwargs: tuple[int, int]:
    ...

# this should be generated
def convert_dict(m: list[str]) -> dict[str, tuple[int, int]]:
    assert len(m) % 3 == 0
   ...

[anntzer]

This looks like a different request? What I had in mind was --foo 1 --bar 2 mapping to {"foo": 1, "bar": 2}. In particular, for your case, it would make sense to have two flags both with these semantics (--dict1 foo 1 bar 2 --dict2 quux 3 mapping to {"dict1": {"foo": 1, "bar": 2}, "dict2": {"quux": 3}}), whereas with the original interpretation, the point would be that any unknown flag would end up in kwargs.

I think your request can be more generally thought of as a "custom parser that takes more than one parameter", i.e. something like

class MyDict(dict): pass  # just a marker class
def mydictparser(*args): return dict(zip(args[::2], map(float, args[1::2])))
def main(kws: MyDict): ...
defopt.run(main, parsers={MyDict: mydictparser})

which is not supported now (currently custom parsers all take a single argument) but seems reasonable to have (well, as usual someone needs to write the implementation :-)) -- we could just introspect the signature of the custom parser...

I guess supporting variadic custom parsers is also linked to dataclass support (#82 (comment)), as these basically could behave like variadic parsers (except for the tricky question of whether they should introduce flags or be positional...)