The CTW2019 and CTW2020's dvc.yaml config currently uses a list of models and inputs. This was a temporary workaround when the solution was a work in progress. The final goal is to move all models and input types/variants specified in params.yaml file. I think this still needs to be talked through. A possible solution is something like this:
# dvc.yaml
benchmark:
desc: Determine best hyper-parameters for algorithm on several CVs
matrix:
split: ${splits}
model: # <--- We remove this
- name: XGBRegressor
input: ${path.data}/prepared/ctw2020.flatten.pkl
- name: RandomForestRegressor
input: ${path.data}/prepared/ctw2020.flatten.pkl
- name: Arnold2018DeepModel
input: ${path.data}/prepared/ctw2020.flatten.pkl
- name: Arnold2019SoundingModel
input: ${path.data}/prepared/ctw2020.pkl
- name: Pirnat2022Pirnat1G
input: ${path.data}/prepared/ctw2020.pkl
- name: Cerar2021Localization
input: ${path.data}/prepared/ctw2020.pkl
...
# params.yaml
models:
Arnold2018DeepModel:
input_type: bchw # <--- and introduce this (possible values something like: flatten, bchw, chwc)
...
It's crucial to update the benchmark.py script to match the changes in the dvc.yaml and params.yaml configs.
The CTW2019 and CTW2020's
dvc.yamlconfig currently uses a list of models and inputs. This was a temporary workaround when the solution was a work in progress. The final goal is to move all models and input types/variants specified inparams.yamlfile. I think this still needs to be talked through. A possible solution is something like this:It's crucial to update the
benchmark.pyscript to match the changes in thedvc.yamlandparams.yamlconfigs.