Daqinterface develop instructions

A brief overview

{{toc}}

DAQInterface is a set of Python and Bash scripts which make it easy to start and run an existing artdaq-based DAQ system. Specifically, this means the ability to select how many of each type of artdaq process (BoardReaders, EventBuilders, DataLoggers, Dispatchers and RoutingMasters) to use in a running DAQ system, as well as what hosts they should run on. Additionally, DAQInterface provides built-in support for sending transitions via XML-RPC to artdaq processes at the command line as well as the saving of metadata related to a run (such as the version of artdaq used, the location of the artdaq logfiles, etc.).

Getting started

Getting DAQInterface to perform runs takes about 10 minutes, assuming you already have an installation of the artdaq-demo package which is based on artdaq v3. If you don’t already have an artdaq-demo installation, you can just execute the following:

wget https://github.com/art-daq/artdaq-demo/repository/revisions/develop/raw/tools/quick-mrb-start.sh
chmod +x quick-mrb-start.sh
./quick-mrb-start.sh --tag=v3_07_02

Using other artdaq-based packages besides artdaq-demo with DAQInterface is covered later in this wiki, though you’re highly encouraged to go through the instructions which cover DAQInterface control of artdaq-demo. If you’re stuck after reading the documentation, please contact John Freeman at jcfree@fnal.gov, not just to get help but also to inform him how the instructions can be improved for the end user.

To get DAQInterface from the head of its develop branch, you’ll want to check out its git repository in an appropriate location:

cd <appropriate location>
git clone https://github.com/art-daq/artdaq_daqinterface.git
cd artdaq-utilities-daqinterface
git checkout develop

…where henceforth we’ll refer to the full path of the artdaq-utilities-daqinterface repo you just checked out as “$ARTDAQ_DAQINTERFACE_DIR”; this keeps things consistent with how you’d interact with DAQInterface if you were using it as a versioned ups product. To get the shell to play along, just execute

export ARTDAQ_DAQINTERFACE_DIR=<appropriate_location>/artdaq-utilities-daqinterface

Now, let’s create a DAQInterface work area:

mkdir $HOME/DAQInterface    # You can, of course, make a directory with another name if you like
cd $HOME/DAQInterface
cp $ARTDAQ_DAQINTERFACE_DIR/bin/mock_ups_setup.sh .
cp $ARTDAQ_DAQINTERFACE_DIR/docs/user_sourcefile_example .
cp $ARTDAQ_DAQINTERFACE_DIR/docs/settings_example .
cp $ARTDAQ_DAQINTERFACE_DIR/docs/known_boardreaders_list_example .
cp $ARTDAQ_DAQINTERFACE_DIR/docs/boot.txt .

Since we can’t set up DAQInterface as a ups product if we’re simply working with the head of its develop branch, we’ll want to use the mock_ups_setup.sh script to mimic ups behavior. If you open up your copy of it, you’ll see the following at the top:

export ARTDAQ_DAQINTERFACE_DIR="directory_of_checked-out_DAQInterface_git_repository_needs_to_be_defined"

And what you’ll want to do is edit this line so that ARTDAQ_DAQINTERFACE_DIR does, in fact, point to the artdaq-utilities-daqinterface repo, like you did interactively above in the shell.

Since DAQInterface doesn’t know the experiment-specific requirements of its user, we’ll want to create a file to source which sets up an environment appropriate for our needs. In our area, this file is “user_sourcefile_example”. Let’s open it up, and look at the first two lines:

export DAQINTERFACE_SETTINGS=$ARTDAQ_DAQINTERFACE_DIR/docs/settings_example
export DAQINTERFACE_KNOWN_BOARDREADERS_LIST=$ARTDAQ_DAQINTERFACE_DIR/docs/known_boardreaders_list_example

What you’ll want to do is change this first line so that for the DAQINTERFACE_SETTINGS variable value we replace the “$ARTDAQ_DAQINTERFACE_DIR/docs” with the name of the directory into which we copied the settings_example file. Then do the same for the DAQINTERFACE_KNOWN_BOARDREADERS_LIST variable on the second line. Once you’ve done that, let’s open the settings_example file itself, and modify the line

productsdir_for_bash_scripts: $HOME/path_to_product_dir_not_yet_defined/products

so that the value of “productsdir_for_bash_scripts” points to a products directory which contains xmlrpc_c (for sending messages to DAQInterface), root (to allow the execution of scripts which crosscheck the *.root files output by the DAQ), and fhiclcpp (so DAQInterface can reformat the FHiCL documents it sends to the artdaq processes). Then modify the lines

log_directory: /tmp
record_directory: $HOME/run_records

so they refer to the directories in which you’d like artdaq to output its logfiles (log_directory) and to save records of what happened during each run (record_directory). It’s recommended that you not leave the log directory as “/tmp”, otherwise your logfiles will most likely be periodically removed. Of course, make sure the directories you set here actually exist, and that you’ve set the directory permissions appropriately (e.g., if other users will also be writing their logfiles and their run records to those areas, you’d want to make sure they’d have proper access).

The last file we’ll edit is the boot.txt file. This file will be passed to DAQInterface on the boot transition (i.e., the transition in which DAQInterface launches the artdaq processes). Take its top line:

DAQ setup script: name_of_DAQ_setup_script_not_yet_defined

…and replace name_of_DAQ_setup_script_not_yet_defined with the fully-qualified filename of the script you want DAQInterface to source before it tries to launch artdaq processes. E.g., if you installed artdaq-demo with the quick-mrb-start.sh script in $HOME/artdaq-demo-installation, you’d want to replace “name_of_DAQ_setup_script_not_yet_defined” with “$HOME/artdaq-demo-installation/setupARTDAQDEMO”.

You’ll also see that hosts (all localhost) have been defined for two EventBuilders, a DataLogger and a Dispatcher; we’ll leave them like that for now.

A first walkthrough

Now that we’ve made the needed modifications to the files in our area, it’s time to set up the environment for DAQInterface to run. In the terminal, type

source ./mock_ups_setup.sh
export DAQINTERFACE_USER_SOURCEFILE=$PWD/user_sourcefile_example
source $ARTDAQ_DAQINTERFACE_DIR/source_me

…where for the future, you may want to consider putting all those commands in a single file to source, since you need to do this whenever you log in and want to use DAQInterface. You’ll see something like the following:

* The command was successful
* To launch, just type "DAQInterface &" (excluding quotes, ampersand optional)
* Output will be logged in /tmp/daqinterface_jcfree/DAQInterface_partition<N>.log, where <N> 
  is the partition the DAQInterface instance is on
* You're encouraged to read the DAQInterface Manual at https://github.com/art-daq/artdaq_utilities/wiki/Artdaq-daqinterface

so then we type

DAQInterface &

and you’ll see one of two possible outputs. Assuming there isn’t already a DAQInterface running on the default port DAQInterface listens on, you’ll see something like the following:

Wed Nov 20 16:04:54 CST 2019: DAQInterface in partition 0 launched and now in "stopped" state, listening on port 10000

in which case the launch was successful, and you can skip down to the next paragraph. However, if there IS a DAQInterface already running on your host, you’ll see:

There already appears to be a DAQInterface instance running on the requested partition number (0); please either kill the instance (if it's
yours) or use a different partition. Run "listdaqinterfaces.sh" for more info.

where if you run listdaqinterfaces.sh you’ll see something like:

DAQInterface instance was launched at 11:08 by jcfree in partition 0 listening on port 10000

In your environment, there’s an environment variable called DAQINTERFACE_PARTITION_NUMBER which is used when DAQInterface is launched to determine which port to listen on. Partition numbers count up sequentially from 0, and only one DAQInterface instance can be used per partition. The default value of DAQINTERFACE_PARTITION_NUMBER is 0; if there’s already a DAQInterface instance running on this partition, your options are (1) to launch DAQInterface on an unused partition, or (2) to kill the DAQInterface instance using partition 0, assuming it’s controlled by your user account and you know it’s not in active use. To launch DAQInterface in an unused partition, you could just do the following, e.g.:

export DAQINTERFACE_PARTITION_NUMBER=1
DAQInterface &

and to kill the existing DAQInterface on partition 0 and launch your new one, you could do

kill_daqinterface_on_partition.sh 0
DAQInterface &

…where if the kill_daqinterface_on_partition.sh command appears to hang, hit Ctrl-c and run it again with the option “—force” added.

DAQInterface is now ready to receive transition commands. Now type the following:

just_do_it.sh ./boot.txt 0 --config "demo" --comps "component01 component02"

This command can be understood to mean, “Give DAQInterface the file ./boot.txt which tells it how many non-BoardReader artdaq processes to launch and where to launch them, initialize the processes after they’re launched with the FHiCL documents found in the configuration called demo, and in particular, initialize two BoardReaders with FHiCL documents called component01.fcl and component02.fcl”. After executing the command, give it a few moments, and you should eventually see something like the following:

Thu Nov 21 14:36:35 CST 2019: START transition complete for run 3255
Will acquire data until Ctrl-c is hit

DAQInterface is now in the “running” state. We’ll step through the commands executed automatically by the script in a moment, but essentially what we’ve done is launched a system running two ToySimulator fragment generators in two BoardReader processes, where events are assembled by two EventBuilders and sent to a DataLogger which saves them to disk and sends them to a Dispatcher. To stop datataking, hit Ctrl-c from the terminal running just_do_it.sh. Momentarily, you should return to the command line, and in the terminal running DAQInterface, you may need to scroll up a bit but you should see something like:

Thu Nov 21 14:38:18 CST 2019: TERMINATE transition complete
Process manager logfiles (if applicable): 

Disregard this last line, as we’re using a process management method which doesn’t send output to a single file (more on process management methods later).

A second walkthrough

Whereas just_do_it.sh strung multiple transitions together in order to make life simpler for the user, for educational purposes it’s useful to step through running DAQInterface one transition at a time. In this section, we’ll do just that, as well as provide a fuller explanation for the output produced during a run. While not required, you may want to execute the commands from another terminal as it makes the interface a bit cleaner; in order to do this, open a second terminal, cd into the “DAQInterface” working directory you created earlier, and do the following:

cd $HOME/DAQInterface   # Or whatever your DAQInterface working directory is
source ./mock_ups_setup.sh
export DAQINTERFACE_USER_SOURCEFILE=$PWD/user_sourcefile_example
source $ARTDAQ_DAQINTERFACE_DIR/source_me

…assuming you haven’t put the last three commands in a source-able file as recommended in the previous section. Now that you’ve set up the environment, assuming you launched DAQInterface in partition 0, run the following:

status.sh

or, if you launched it in another partition, you’ll want to set the DAQINTERFACE_PARTITION_NUMBER environment variable accordingly as well. E.g., if you launched DAQInterface on partition 1, you’d want to do:

export DAQINTERFACE_PARTITION_NUMBER=1 
status.sh 

Either way, you should see this:

Result:

String: 'stopped'

Essentially, status.sh queries the DAQInterface state machine and prints DAQInterface’s state to the screen. just_do_it.sh uses this script to ensure that the transitions it sends DAQInterface went through successfully. We’re now in the “stopped” state of DAQInterface - this is the ground state, and means that no artdaq processes exist. In order to launch processes, we send the “boot” transition. To send individual transitions, we use the send_transition.sh script. However, before we boot the system, we need to specify which components (fragment generators) we want to use. The allowed fragment generators will be listed by DAQInterface if you issue it the following command:

listdaqcomps.sh

where the output, in the terminal in which DAQInterface was launched, will look something like this:

# of components found in listdaqcomps call: 30
component01 (runs on localhost)
component02 (runs on localhost)

..etc. These lines refer to ToySimulator instances designed to run on the current host; in a real experiment you’d expect a more complex listing. They are defined in the file referred to by the DAQINTERFACE_KNOWN_BOARDREADERS_LIST environment variable; in our case this is known_boardreaders_list_example. Let’s select two components (although we could run with just one, or all of them, if we wanted):

setdaqcomps.sh component01 component02

…at the end of the output when you execute this is “Result: Nil”; this is expected, and occurs whenever you send DAQInterface this command, or any of the upcoming transitions. Now we can send the boot transition:

send_transition.sh boot boot.txt

In the terminal running DAQInterface, you’ll see something like this:

Thu Dec 12 15:28:54 CST 2019: BOOT transition underway

On randomly selected node (localhost), will confirm that the DAQ setup script 
~/artdaq-demo_v3_07_01/setupARTDAQDEMO
doesn't return a nonzero value when sourced...done (3.9 seconds).

Launching the artdaq processes

Checking that processes are up (check 1 of a max of 5 checks)...found 6 of 6 processes.
All processes appear to be up

artdaq_mfextensions v1_03_08, e17:prof:s85, appears to be available; if
windowing is supported on your host you should see the messageviewer
window pop up momentarily

Determining logfiles associated with the artdaq processes...done (0.1 seconds).

Thu Dec 12 15:29:16 CST 2019: BOOT transition complete

If you’re sending transitions in the same terminal as DAQInterface is running, hit Enter to return to the command line. The boot.txt file which you edited earlier is used by DAQInterface during the boot transition to determine where to find the script to source before it tries to launch the processes (known as the “DAQ setup script”), and then how many EventBuilder, DataLogger, Dispatcher and RoutingMaster processes to launch and where to launch them. The BoardReader processes which get launched are defined when setdaqcomps.sh is called, combined with the information provided by known_boardreaders_list_example. What the above output tells us is that DAQInterface has successfully sourced the DAQ setup script on one of the nodes it’ll use during the run, that it’s successfully launched the artdaq processes, and that since the version of artdaq_mfextensions needed by the MessageViewer application is available, assuming your terminal’s display settings are set correctly, a window will pop up which will print messages sent by the artdaq processes. Finally, it determines the names each artdaq process’s logfile; this information will be saved later by DAQInterface. You can see the statuses of the individual artdaq processes if you execute “artdaq_process_info.sh”.

Now that the processes are alive, we’ll configure them with FHiCL documents. Simply execute the following:

send_transition.sh config demo

“demo” is the name of the configuration which defines the FHiCL documents used to initialize the artdaq processes. Technically speaking, it refers to the subdirectory which can be found in the $ARTDAQ_DAQINTERFACE_DIR/simple_test_config directory, and which contains the needed FHiCL documents. Note that you can have multiple arguments to config, each argument being the name of a different subdirectory; in this case each argument is referred to as a “subconfiguration”, and by convention the name of the configuration is the set of subconfigurations listed alphabetically separated from one another by a space. If you use subconfigurations, the filenames must be unique across all subconfigurations; e.g., if you use a subconfiguration called “eventbuilders” and a subconfiguration called “dataloggers”, you can’t have a file called metrics.fcl appear in each subconfiguration- otherwise DAQInterface will throw an exception. It’s possible to change the way DAQInterface interprets a (sub)configuration name by editing the user source file, either by switching it to look at another directory on the filesystem or getting it to use an instance of artdaq-database, discussed in the advanced links at the bottom of this wiki. If you wish to see the list of available configurations, just type

listconfigs.sh

and you’ll see something like the following:

Available configurations: 
ascii_simulator_example
combined_eb_and_dl
config_includes
demo
demo_largesystem
eventbuilder_diskwriting
file_closing_example
mediumsystem_with_routing_master
multiple_art_processes_example
multiple_dataloggers
request_based_dataflow_example
routing_master_example
simple_subsystems
subconfigs
subrun_example

See file "/tmp/listconfigs_jcfree.txt" for saved record of the above configurations

Please note that for the time being, the optional
max_configurations_to_list variable which may be set in
~/scratch/DAQInterface_for_updated_documentation/settings_example
is only applicable when working with the database

Most of these configurations are intended for developer use; we’ll be working with “demo”. Scrolling back up to see the end of the output from the config transition you’ve sent, you should see “All artdaq processes returned ”Success”.“; if this isn’t the case, DAQInterface will print out each individual process’s returned string for you for troubleshooting purposes.

Along with the new logfiles, you’ll also find in the “/tmp” directory a directory called “run_record_attempted_/”, where “” is your username stored in the environment variable $USER, and is the moment the directory of the same name was created. In here, what you’ll find are the following:

-The FHiCL documents which are used to initialize the artdaq processes during the config transition
-The DAQInterface boot file, boot.txt
-The metadata file, metadata.txt
-A copy of the script which DAQInterface sourced before launching the artdaq processes, setup.txt
-The list of possible boardreaders DAQInterface used, known_boardreaders_list.txt
-The list of process ranks, ranks.txt. This contains relatively low-level information about the artdaq processes which were actually used in the run, including the host they ran on, the port they listened for commands on, and their “rank”. Some diagnostic messages artdaq produces refer to a process’s “rank”, and this list will translate a rank to the actual process in question.
-The settings file used when DAQInterface was launched, settings.txt
-A file containing the values of all the environment variables used to control DAQInterface’s behavior, environment.txt

If something goes wrong during configuration, the contents of /tmp/run_record_attempted*USER>/ can potentially be useful for diagnostic purposes; the same information can also be found in /tmp/run_record_attempted*/, but be aware that while this second directory path is easier to find/type, it gets clobbered the next time that a configuration takes place on the partition. If all goes well, the attempted run record gets moved to the official run records directory on the start transition, covered next. Click on “Show” for details on the metadata file:

{{collapse()
The metadata file which gets saved in the run records directory (defined in the settings file as described earlier in this wiki) is formatted like the following example. The meaning of parameters will be explained below:

Config name: demo
Component #0: component01
Component #1: component02
DAQInterface directory: host01:~/DAQInterface
DAQInterface logfile: host01:/tmp/daqinterface_jcfree/DAQInterface_partition1.log

# Two possible sets of fields provided below for code info, depending on if a git repo was available: 
# <git commit hash> <LoCs added on top of commit> <LoCs removed on top of commit> <git commit comment> <git commit time> <git branch> <BuildInfo build time (if available)> <BuildInfo version (if available)>
# <package version> <BuildInfo build time (if available)> <BuildInfo version (if available)>

DAQInterface commit/version: b809b5f5eca2dfe4f64f0a6f89817f96c96af5cc 0 0 "JCF: Issue #23445: allow for underscores AND hyphe..." "Fri Oct 18 16:04:39 2019 -0500" "develop" "time from BuildInfo undetermined" "version from BuildInfo undetermined"

artdaq commit/version: 54a6d13f931f1f7f2895e1379018b66915d771a8 0 0 "Add s83 to qualifier list " "Mon Sep 23 14:27:07 2019 -0500" "(detached from v3_06_01)" "04-Oct-2019 20:53:50 UTC" "artdaq-v3_06_01"

artdaq-core commit/version: v3_05_03 "23-Sep-2019 14:24:59 UTC" "v3_05_03"

artdaq-core-demo commit/version: 45cd83987c85db67ab6b2ca35e966917c6e38cf7 0 0 "Add s83 to product_deps " "Mon Sep 23 14:49:45 2019 -0500" "(detached from v1_06_16)" "04-Oct-2019 20:53:46 UTC" "artdaq-v3_06_01"

artdaq-demo commit/version: 4742bb9fdbc4f50c526ffcd43a7b77bab9ac29a7 0 1 "Add s83 to product_deps " "Mon Sep 23 14:50:57 2019 -0500" "(detached from v3_06_01)" "04-Oct-2019 20:53:48 UTC" "artdaq-v3_06_01"

artdaq-utilities commit/version: v1_05_01 "24-Sep-2019 14:40:44 UTC" "v1_05_01"

process management method: direct


process manager logfiles: 

boardreader logfiles: 
host02:~/daqlogs/component01-host02-11101/component01-host02-11101-20191018161757-19250.log
host01:~/daqlogs/component02-host01-11100/component02-host01-11100-20191018161757-433832.log

eventbuilder logfiles: 
host02.fnal.gov:~/daqlogs/EventBuilder1-host02-11102/EventBuilder1-host02-11102-20191018161757-19355.log

datalogger logfiles: 
host01:~/daqlogs/DataLogger1-host01-11103/DataLogger1-host01-11103-20191018161757-433833.log

dispatcher logfiles: 

routingmaster logfiles: 

• Config name: This is the name of the configuration which got passed during the config transition
• Component #: The Nth component (BoardReader w/ fragment generator defined) selected for the run
• DAQInterface directory The directory out of which DAQInterface was run
• DAQInterface logfile The file where DAQInterface’s output to stderr and stdout gets saved
• commit/version:
  • If the package’s git repo was available: its git commit hash, lines of code added on top of the commit, lines of code removed on top of the commit, the beginning of the git commit comment, the git commit time (in timezone the commit was made), and then the time (UTC) and version of the build from the package’s BuildInfo module (if available)
  • If the package’s git repo wasn’t available: the version of the package used as found in ups, and then the time (UTC) and version of the build from the package’s BuildInfo module (if available)
    Packages which get listed in this metadata file can be selected in the settings file, referred to by the environment variable $DAQINTERFACE_SETTINGS.
• process management method: The technique DAQInterface used to launch/kill artdaq processes. Defined by the $DAQINTERFACE_PROCESS_MANAGEMENT_METHOD environment variable before DAQInterface is launched (defaults to “direct”).
• process manager log files: If the pmt technique is used to launch and kill processes, the logfile which captures all the stdout and stderr of artdaq processes is provided here. If the direct technique is used, this will be empty.
• logfiles: The names of every individual process type’s output logfile (one per process), each one prefaced by the host the process in question ran on.

The above variables are the ones which appear in the metadata file during the config transition. However, other variables related to the actual running will appear as you go through the start and stop transitions described below; by the time the metadata file has been finished being written to, something like the following will have been added:

DAQInterface start time: Fri Oct 18 21:18:23 UTC 2019

DAQInterface stop time: Fri Oct 18 21:18:36 UTC 2019

}}

Now let’s issue the start transition:

send_transition.sh start

and then DAQInterface will report

Wed Dec 21 16:22:29 CST 2016: START transition complete for run 2

Now we’re in the “running” state, as you can confirm if you issue the status.sh command. Note that the run # is deduced from the run records directory - it’s a value one higher than the most recently saved run in the run records directory. If you look in the run records directory you’ll now see that the contents of /tmp/run_record_attempted_/ have been copied into the subdirectory called “2/”, or whatever the number of the run you just started is.

Stopping datataking and putting DAQInterface in the “ready” state is quite simple:

send_transition.sh stop

at which point, you can either start a new run, or you can send the artdaq processes the “shutdown” command and then kill them, via the terminate transition:

send_transition.sh terminate

You can confirm that you’ve created a *.root file from the run by typing show_rootfiles_for_run.sh .

If you wish to repeat a run you’ve already taken without needing to worry about remembering which configuration you used, which boardreaders, etc., you can use the just_repeat_run.sh script, like follows:

just_repeat_run.sh <run number> 0 

which will use the same boot file, the same configuration and the same boardreaders as were used during the run referred to by . Be aware that if there’s a mismatch between the code used during (as determined by the info found in the metadata.txt file for ) and the code which currently exists in the installation referred to by DAQ setup script in the boot file (e.g., commits were made since was performed), then just_repeat_run.sh will refuse to proceed; you can override this behavior by passing it the —nostrict argument. Similar behavior occurs if you’re running DAQInterface on a different node than it was run on for , or if DAQINTERFACE_KNOWN_BOARDREADERS_LIST is pointing to a different known boardreaders list.

DAQInterface for your experiment

If you’ve reached this point in the documentation, you’re now ready to prepare DAQInterface to work not with artdaq-demo, but with your own experiment’s artdaq-based software. You’ll want to make copies of the settings file and boot file you’ve been working with and make the appropriate edits; how to do so is described in [[daqinterface_develop_settings_file_reference | The settings file reference]] and [[daqinterface_develop_boot_file_reference | The boot file reference]]. You’ll also want to switch to using the artdaq-database as the source of your configurations rather than the local directory in the DAQInterface package; making this switch is described in Changing the source of the configurations. DAQInterface has a default set of rules as to when to automatically end a run if an artdaq process dies or enters the Error state; to override this set of rules, take a look at [[Defining which processes are critical to a run]].

If you wish to use multiple subsystems in your experiment, the way to inform DAQInterface is described [[Daqinterface develop subsystem instructions | here]].

If you wish to use fragment generators which produce either zero or more than one fragment per event, make sure the FHiCL document in your configuration for that fragment includes the line “generated_fragments_per_event: ”, where is to be replaced by an integer denoting the number of fragments per event. This way, DAQInterface can correctly modify the eventbuilder FHiCL documents before sending them to eventbuilders so that they know how many fragments make up an event.

To change the underlying method by which DAQInterface launches processes, set the DAQINTERFACE_PROCESS_MANAGEMENT_METHOD variable accordingly to one of the following values:

• direct means DAQInterface directly launches the “boardreader”, “eventbuilder”, etc. executables from the artdaq package. This is the default value.
• pmt means DAQInterface uses the artdaq_mpich_plugin package’s pmt.rb script to launch processes. Practically speaking, the difference between this approach and “direct” is that with “pmt”, if an artdaq process dies when DAQInterface is in the “running” state it ends the run, while with “direct”, you have the option of not letting this happen (click on “Defining which processes are critical to a run” a few lines up). Additionally, a feature of pmt.rb is that the output of all the artdaq process is collected in a convenient single log file; this feature is not available in “direct” mode, so each artdaq process’s individual logfile must be examined.
• external_run_control means DAQInterface doesn’t launch or control processes at all, but instead defers that behavior to some other entity. Currently this option is used by ProtoDUNE. Using it correctly requires providing DAQInterface with information it doesn’t typically need; for details contact John Freeman at jcfree@fnal.gov.

When you call the setdaqcomps.sh command, DAQInterface will look in the file referred to by the DAQINTERFACE_KNOWN_BOARDREADERS_LIST environment variable and use the file’s contents to determine the host and (optionally) the port on which to launch the BoardReaders; the corresponding information for the other artdaq process types is provided in the boot file. You’ll want to make a copy of the $DAQINTERFACE_KNOWN_BOARDREADERS_LIST file we’ve been using, and edit it, since you’ll want to give your experiment’s fragment generators more descriptive names than “component01”, “component02”, etc. E.g. if you want to use the name tpcreceiver02, then you’ll want to both make sure that a file called tpcreceiver02.fcl exists in the configuration you use, and also that there’s a corresponding row in the file $DAQINTERFACE_KNOWN_BOARDREADERS_LIST which looks like this:

tpcreceiver02  madeuphost.fnal.gov

where of course you’ll want to use a host which is appropriate for your purposes. Also make sure to edit the user_sourcefile_example to set DAQINTERFACE_KNOWN_BOARDREADERS_LIST correctly.

Additional options exist for controlling the behavior of a BoardReader in the $DAQINTERFACE_KNOWN_BOARDREADERS_LIST file. The fields (the first two of which you’ve already seen) include:

1. BoardReader label
2. Host BoardReader exists on
3. Port BoardReader listens on for DAQInterface commands ( “–1” means let DAQInterface figure this out)
4. Subsystem BoardReader exists in (set to “1” unless you’re using multiple subsystems)
5. Cores BoardReader is allowed to use (if this isn’t set, all cores can be used, otherwise see last comment in https://cdcvs.fnal.gov/redmine/issues/22258 for how to set this)
6. Any arbitrary command, in quotes, to execute before BoardReader process launch (only meaningful if $DAQINTERFACE_PROCESS_MANAGEMENT_METHOD is set to “direct”)

So, e.g.,

tpcreceiver02 madeuphost.fnal.gov -1 1 0-15 "valgrind --tool=callgrind "

means “Run tpcreceiver02 on madeuphost.fnal.gov; let DAQInterface decide which port to send it commands on. tpcreceiver02 is in subsystem 1, can use the first 16 cores on the host, and we want to run valgrind on it”

Note that to set the Nth field for a BoardReader, you’ll also need to set fields 1 through N-1.

DAQInterface contains a mechanism by which certain functions which default to no-ops can be overridden in an experiment-defined module called “daqinterface_overrides_for_experiment.py”. These include “perform_periodic_action_base”, which executes a function in a loop roughly once a second, and “start_datataking_base”/”stop_datataking_base”, which execute just after/before the artdaq processes start/stop. It’s recommended you get in touch with a member of the artdaq team if you wish to perform an override.

At this point in the manual, you’ve learned about many environment variables used to control DAQInterface. While you’re encouraged to bookmark this manual in your browser and refer back to it as needed, for a quick refresher as to what the most important environment variables are and what they do, just type

DAQInterface --help

at the command line.

Examining your output

First, you can look at a quick summary of your run with the following command:

show_recent_runs.sh <N, N being the last N runs for which you want to see a summary>

so to just see your most recent run, you can execute “show_recent_runs.sh 1” and then you’ll see something like:

Run 2 (Jul 25 12:53) : demo                          : component01 component02 

Assuming all the transitions went off without a problem

The DAQInterface package provides a couple of convenient tools for rapidly examining the results of your run after it’s complete. As described earlier, the run records directory provides a great deal of information on the details of the run, but to look for more detail the following scripts can be used.

To look at the logfile for a specific artdaq process in your run, you can do the following:

show_logfile_for_process.sh <run number> <process label - e.g., "component01" or "EventBuilder1"> <open file (0 or 1)>

For troubleshooting, you can also do

show_warnings_for_run.sh <run number>

…will show all MessageFacility messages of severity level Warning or Error in the artdaq logfiles - with the caveat that some warning messages issued by artdaq processes appear in every run, are considered background noise, and as such don’t get output by this script, even though they can still be found in the logfiles. Also be aware that warnings/errors produced by the art processes owned by artdaq processes aren’t shown here, since they don’t make it into the artdaq logfiles.

If your run was performed with DAQINTERFACE_PROCESS_MANAGEMENT_METHOD set to “pmt”, then since all the artdaq process’s output gets sent to a single file, you can run this command:

show_logfile_for_run.sh <run number> <open file (0 or 1)>

…will show the logfile for the run number given in the first argument, and will open the file for you (or not) depending on whether the second argument is 1 or 0.

It’s also possible to take a look at the root files produced during the run. If you execute:

show_rootfiles_for_run.sh <run number>

the script will look at every artdaq process on every host which (tried) writing out root file(s) during the run, figure out where the root file(s) was/were written, and then list them. You can add an optional second argument which is the number of events per root file you want shown in detail. Note that this per-event feature will work only if at least one of the following is satisfied:

• On the host in question, an experiment-defined alias/executable called “rawEventDump”, defined by the experiment, happens to be available
• The DAQ setup script saved as “setup.txt” in the run record, when sourced, produces an alias called “rawEventDump” (as is the case in, e.g., artdaq-demo v3_06_00’s setupARTDAQDEMO script)
• The DAQ setup script is able to successfully set up artdaq

Please also note that you can always take a look at previous runs by opening the DAQInterface logfile, found in the directory referred to by the $DAQINTERFACE_LOGDIR environment variable and of the form “DAQInterface_partition.log”

Finally, you can even read and set TRACE levels of the artdaq processes you’re controlling. To read them, you can call

trace_get.sh

and for each process, an ASCII file will be written to the /tmp directory containing the level, with a name of the format:

/tmp/trace_get_<process label>_${USER}_partition${DAQINTERFACE_PARTITION_NUMBER}.txt

where could be, e.g., EventBuilder1, component01, etc.

You can also set the levels via trace_set.sh; the arguments here would be

trace_set.sh <level name> <masktype - T, S, or M> <maskval, expressed in quotes>

If there was a failure during the config transition

In this case, the scripts described above won’t work since “run number” isn’t a meaningful concept in this case - you weren’t able to begin the run at all, and there’s nothing in the run records directory. However, DAQInterface does save a temporary directory which contains the files that would have been placed in the run record had the run began; those files can be found in this area: /tmp/run_record_attempted*your username>/. Note that this gets clobbered every time a config is performed on that partition; for a permanent but harder-to-type record of the failed transition, you can look at /tmp/run_record_attempted*/.

If there was a failure during the boot transition

This case is the trickiest to debug, since an artdaq process which doesn’t launch doesn’t have a logfile. You’ll see something like this on the screen, though you may need to scroll up a bit to see it:

The following desired artdaq processes failed to launch:

…followed by the list of processes which didn’t launch. Often it’s the case that all the processes associated with a certain node are listed, perhaps because of a networking issue (firewall, etc.) One or two things to try: (1) if you’re running with DAQINTERFACE_PROCESS_MANAGEMENT_METHOD=“direct”, on the node(s) where processes failed to launch, take a look at /tmp/launch_attempt__partition; this file will contain both stdout and stderr from all artdaq processes on the node. (2) trying to boot again, but setting the “debug level” in the boot file to “4”, its most verbose level.