Reorganized the integration test parameters that developers use to specify a DUNE DAQ configuration

[bieryAtFnal]

... The goal is to help make integtests easier to write and understand.

Description

These changes are intended for fddaq-v5.7.0.

The parameters that developers use to tell the integrationtest infrastructure about the characteristics of a requested regression/integration test have historically been a little confusing (at least to me).

The goal of the changes in this PR and the related ones in several other repositories is to make integration tests easier to understand.

A first example of how these changes attempt to do this is the following:

• instead of
  • using a special global variable (object_databases) to request that the DUNE-DAQ configuration be generated by the integrationtest infrastructure, and
  • using a certain parameter in a data structure to request/specify that a pre-defined DUNE-DAQ configuration should be used in the integtest
• these changes introduce the use of separate data structures for generated and pre-defined DUNE-DAQ configurations.

The integtest_params_for_generated_dunedaq_config Python dataclass has been introduced to request a generated DUNE-DAQ configuration, and it has fields that are appropriate for such generation.

The integtest_params_for_predefined_dunedaq_config Python dataclass has been introduced to specify a pre-defined DUNE-DAQ configuration, and it has field(s) that are tailored to pre-defined DUNE-DAQ configurations.

Examples of the use of these dataclasses are available in the updated integtests in various repositories. Here is a snippet of code from the minimal_system_quick_test that requests/specifies a generated DUNE-DAQ configuration:

conf_dict = data_classes.integtest_params_for_generated_dunedaq_config()
conf_dict.object_databases = ["config/daqsystemtest/integrationtest-objects.data.xml"]
conf_dict.dro_map_config.n_streams = number_of_data_producers
conf_dict.op_env = "integtest"
conf_dict.config_session_name = "minimal"
conf_dict.tpg_enabled = False

Here is a snippet of code from the example_system_test that requests/specifies that a pre-defined DUNE-DAQ configuration be used:

common_config_obj = data_classes.integtest_params_for_predefined_dunedaq_config()
common_config_obj.op_env = "test"
common_config_obj.predefined_config_db = (
    os.path.dirname(__file__) + "/../config/daqsystemtest/example-configs.data.xml"
)

onebyone_local_conf = copy.deepcopy(common_config_obj)
onebyone_local_conf.config_session_name = "local-1x1-config"

The changes in this repository are primarily related to the introduction of these two new dataclasses and the switch to using them. The changes in the daqsystemtest, dfmodules, hsilibs, listrev, snbmodules, and trigger repositories were needed to keep the various integtests in sync with these changes.

Here are the PRs for the changes in other repositories that are correlated with this one:

• Changes to adapt to confgen (integtest config params) improvements daqsystemtest#319
• initial changes to adapt to confgen (integtest config params) improvements dfmodules#489
• initial changes to adapt to confgen (integtest config params) improvements hsilibs#85
• initial changes to adapt to confgen (integtest config params) improvements listrev#123
• initial changes to adapt to confgen (integtest config params) improvements snbmodules#33
• initial changes to adapt to confgen (integtest config params) improvements trigger#432

Here are suggested instructions for testing these changes:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260513_A9 ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b kbiery/confgen_ideas
git clone https://github.com/DUNE-DAQ/dfmodules.git -b kbiery/confgen_ideas
git clone https://github.com/DUNE-DAQ/hsilibs.git -b kbiery/confgen_ideas
git clone https://github.com/DUNE-DAQ/listrev.git -b kbiery/confgen_ideas
git clone https://github.com/DUNE-DAQ/snbmodules.git -b kbiery/confgen_ideas
git clone https://github.com/DUNE-DAQ/trigger.git -b kbiery/confgen_ideas
cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b kbiery/confgen_ideas
cd ..

. ./env.sh
dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh --verbosity 2 -r local --random 5
# and/or
dunedaq_integtest_bundle.sh --verbosity 2 -r all -R 'asiolibs|crtmodules' -x 'iceberg|ehn1' --random 5

Type of change

• Optimization (non-breaking change that improves code/performance)

Testing checklist

• Minimal system quicktest passes (pytest -s minimal_system_quick_test.py)
• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

• Code is commented where needed, particularly in hard-to-understand areas